Thursday, November 27, 2014

Killing subprocess heirarchies with Python

I'm currently working on a distributed Python project that uses flask as a communication interface between nodes. I'm doing all my testing locally and starting instances of the application manually becomes tiresome quickly. I decided to write a test harness to simplify things. Things were going swimmingly until I tried to implement some cleanup functionality to kill all the nodes. Here's why - flask forks off a child process when you start it up:
$ ./app.py 
$ ps | grep app.py
dkudrow    4230    Ss    env/bin/python ~/src/project/app.py
dkudrow    4233    Sl    ~/src/project/env/bin/python ~/src/project/app.py
In my test harness, I start these with subprocess as suggested by the kind folks at Python:
cmd = [ os.getcwd()+'./app.py' ]
proc = subprocess.Popen(cmd)
After sending a SIGTERM like this,
proc.terminate()
I'm left with this mess:
$ ps | grep app.py
dkudrow    4230    Z+      env/bin/python ~/src/project/app.py
dkudrow    4233    Sl+   ~/src/project/env/bin/python ~/src/project/app.py
The first process is killed and becomes a zombie waiting for it's child. Like many things in Python, a simple task has just become irritatingly difficult. Thankfully there is a solution carefully hidden in the documentation.

While subprocess doesn't yet (or may never?) support management of process groups, the os module does. The os.killpg(pgid, signal) method will send a signal to every process in a group. We can mix and match modules here to achieve what we want.

First off, we modify the preexec_fn parameter of the Popen constructor,
proc = subprocess.Popen(cmd, preexec_fn=os.setsid)
preexec_fn can be used to specify a function to call in the new process before cmd is execed. In this case, we call os.setsid() which just calls the setsid system call in the new process. You can refer to the man page for more details but in a few words, setsid creates a new process group and makes the calling process the leader of this group.

We can then call
os.killpg(proc.pid, signal.SIGTERM)
and since proc is a process group leader, all processes in the group (i.e. proc's children) will all recieve the SIGTERM. 

Problem solved.


Saturday, September 28, 2013

Triumph Bonneville Main Jet Swap without removing carbs

This is a quick post illustrating how to change the main jets without having to remove the full carburetor assembly. Being able to swap jets quickly is crucial when modifying your bike and this saves you the pain of pulling half the bike apart.

First, unscrew the four bolts holding the float bowl in place with a small hex key. Make sure to drain the bowl before hand!


With the bowls removed, you can easily see the diffuser and main jet sticking out past the float.



Use an 8mm wrench to unscrew the diffuser. Be aware of the small collar that sits on top of the diffuser. As you are removing the diffuser it may fall out so keep an eye out for it.


Now you can swap the main jet.


And in the timeless words of every workshop manual, "Installation is the reverse of removal."

Sunday, August 18, 2013

PS/2 Keyboard Emulation with Arduino UNO

I have a growing collection of older keyboards that I really like but have no way of using due to the extinct connectors and protocols that they rely on. As an example, my IBM model F uses a 5-pin DIN connector and my Sun Type 4, an 8-pin mini-DIN. Hardly standard stuff. So I came upon the idea of using an Arduino UNO that's been (let's be honest) collecting dust for a while to interface these with a PS/2 keyboard port. I opted not to use USB because I wanted to implement the whole thing myself from scratch and PS/2 seemed like a gentler first step. It's been a long weekend but I'm very pleased with the fruits of my labour.

PS/2

The PS/2 protocol has it's roots in the third iteration of IBM's personal computer. It uses two thirds of a 6-pin mini-DIN connector to transmit +5V, ground, clock and data as shown below:

The protocol is explained brilliantly by these documents but I'll walk through it again here for the sake of continuity. Serial communication occurs between a 'device' (keyboard or mouse) and a 'host' (computer) on a bi-directional data line. Both the clock and data lines interface using open collectors that are held at +5V by a pull-up resistor (this is the idle state). The device or host can send a signal by lowering the impedance and bringing the line to ground. Because of this I'll use the terms 'pull' low and 'release' high when discussing changes in state. Each frame consists of 11 or 12 bits as follows:
  • 1 start bit (low)
  • 8 data bits
  • 1 parity bit (odd - high if there are an even number of 1's in the data)
  • 1 stop bit (high)
  • 1 'ack' bit (only in host-to-device communications)
The device reads from and writes to the data line on the rising edge of the clock while the host does so on the falling edge (it's best to wait half a cell before actually operating though as the signal's rise time can be quite slow). The clock signal is generated by the device except for when the host is inhibiting the device.

Device-to-host communication

The device can send a signal to the host whenever the data and clock lines are high (idle state). The device writes the signal on the rising edge of the clock and the host reads the signal on the falling edge of the clock. In practice however it's best to perform transitions halfway through the clock cell. Below I've illustrated one frame. Each full clock cycle (falling edge to falling edge) must be between 60 and 100 microseconds.


The device write sequence:
  1. Pull data low to set the start bit
  2. Pull the clock low (host reads start bit)
  3. Release the clock
  4. Set first data bit
  5. Pull clock low (host reads data bit)
  6. Repeat 3-5 for remaining data bits and parity bit
  7. Release clock
  8. Release data to set stop bit
  9. Pull clock low (host reads stop bit)
  10. Release clock
The host read sequence:
  1. Wait for the device to pull the clock low
  2. Read the start bit
  3. Wait for the device to release the clock
  4. Wait for the device to pull the clock low
  5. Read the first data bit
  6. repeat 3-5 for each data bit, the parity bit and the stop bit
Host-to-device communication

Since the device controls the clock, the host has to request a clock signal whenever it wants to send a frame. It does so by pulling the clock low and holding it low for 100 microseconds. With the data high, this is the inhibit state which tells the device that it should stop sending information. The host then pulls data low and releases the clock indicating the device should read the start bit of the frame and begin generating clocks.


This image is actually misleading because the the high values are not really generated by either party - it is the resting state of the line. Think of the color as determining who is in control of a signal.

The host write sequence:
  1. Pull the clock low for > 100 microseconds
  2. Pull data low (start bit)
  3. Release clock
  4. Wait for device to pull clock low
  5. Set first data bit
  6. Wait for device to release clock (device reads bit)
  7. Repeat 4-6 for data bits, parity bit and start bit
  8. Wait for device to  pull clock low
  9. Read ack bit
The device read sequence:
  1. Wait for the host to pull the clock low
  2. Wait for the host to release the clock
  3. Read the start bit
  4. Pull clock low (host writes first data bit)
  5. Release clock
  6. Read first data bit
  7. Repeat 4-6 for each data bit, parity bit and stop bit
  8. Pull clock low
  9. Release clock
  10. Pull data line low (set ack bit)
  11. release clock
Arduino

There are a couple of tricks to implementing this protocol on the Arduino. The first is using the internal pull-up resistors. To write a high signal, set the pinMode to INPUT and write a the high signal with digitalWrite. Similarly, to write a low signal, set the pinMode to LOW and again write the signal with digitalWrite. This  ensures that the correct impedance is selected and allows the Arduino to communicate with the collector on the other end. The second trick is, when emulating a device, to connect the host's +5V pin to the Arduino's Vin pin and the host's ground pin to the Arduino's ground. This ensures that the Arduino has the correct reference voltage.


Both the computer and the keyboard connected to the Arduino. This setup is useful for determining the 'handshake' in which the host enables the device and begins listening.


Computer Handshake

When a PS/2 powers up it performs a self check and then sends a 0xAA to indicate that it is ready to interface with the host. What follows is an exchange that culminates in the computer accepting input from the keyboard. This exchange will vary from machine to machine however I stepped through the process with my ASRock motherboard and discovered this sequence. I suspect most machines are similar:

Power up
device: 0xAA (ready)
host:   0xF2 (ID)
device: 0xFA (acknowledge)
device: 0xAB (first byte of ID)
host:   0xED (set LED)
device: 0xFA
host:   0x00 (second byte of set LED, indicates all LED's off)
device: 0xFA
host:   0xF4 (keyboard enabled)
device: 0xFA
host:   0xED
device: 0xFA
host:   0x00
device: 0xFA
host:   0xF3 (set time delay)
device: 0xFA
host:   0x00 (second byte of time delay, not sure what it means)
device: 0xFA

This is the Arduino library I've written to implement emulate PS/2 hosts and devices. This will likely be out of date very soon so check out the github repository here.

//===-- PS2Emu.h - PS/2 Keyboard Host and Device Emulation -----------------===/
//
// Written by Daniel Kudrow (dkudrow@cs.ucsb.edu)
// August 2013
//
// Library to emulate device and hosts of PS/2 protocol
//
// TODO:
// check data/clock state before acting
// handle host inhibition
// listen for host requests to send
// handle host interrupt
//
//===-----------------------------------------------------------------------===/


#ifndef ps2emu_h
#define ps2emu_h

#include "Arduino.h"

class PS2Emu {
  private:
    int clk; // clock pin
    int dat; // data pin
    int cell; // one half of clock cycle time
    // logical high is high-impedance
    void setHigh(int pin) { pinMode(pin, INPUT); digitalWrite(pin, HIGH); }
    // logical low is low-impedance
    void setLow(int pin) { pinMode(pin, OUTPUT); digitalWrite(pin, LOW); }

  public:
    // construct and initialize
    PS2Emu(int c, int d, int t=40) : clk(c), dat(d), cell(t) {
        setHigh(clk);
        setHigh(dat);
    }
    // read as host (from device)
    unsigned char hostRead();
    // read as device (from host)
    unsigned char devRead();
    // write as host (to device)
    void hostWrite(unsigned char);
    // write as device (to host)
    void devWrite(unsigned char);
    // FIXME listen for host request to send
    bool devListen();
};

#endif


//===-- PS2Emu.cpp - PS/2 Keyboard Host and Device Emulation ---------------===/
//
// Written by Daniel Kudrow (dkudrow@cs.ucsb.edu)
// August 2013
//
// Library to emulate device and hosts of PS/2 protocol
//
// TODO:
// check data/clock state before acting
// handle host inhibition
// listen for host requests to send
// handle host interrupt
//
//===-----------------------------------------------------------------------===/

#include "ps2emu.h"

unsigned char PS2Emu::hostRead() {
  unsigned char data = 0x00;
  unsigned char p = 0x01;

  // discard the start bit
  while (digitalRead(clk) == HIGH);
  while (digitalRead(clk) == LOW);  
  
  // read each data bit
  for (int i=0; i<8; i++) {
    while (digitalRead(clk) == HIGH);
    if (digitalRead(dat) == HIGH) {
      data = data | (1 << i);
      p = p ^ 1;
    }
    while (digitalRead(clk) == LOW);
  }
  
  // read the parity bit
  while (digitalRead(clk) == HIGH);
  if (digitalRead(dat) != p) {
    // TODO handle parity bit
  }
  while (digitalRead(clk) == LOW);
  
  // discard the stop bit
  while (digitalRead(clk) == HIGH);
  while (digitalRead(clk) == LOW);
  
  return data;
}

unsigned char PS2Emu::devRead() {
  unsigned char data = 0x00;
  unsigned char p = 0x01;
  
  // wait for the host to release the clock
  while (digitalRead(dat) == HIGH);
  while (digitalRead(clk) == LOW);
  
  // read start bit
  delayMicroseconds(cell/2);
  setLow(clk);
  delayMicroseconds(cell);
  setHigh(clk);
  delayMicroseconds(cell/2);

  // read data bits
  for (int i=0; i<8; i++) {
    if (digitalRead(dat) == HIGH) {
      data = data | (1 << i);
      p = p ^ 1;
    }
    delayMicroseconds(cell/2);
    setLow(clk);    
    delayMicroseconds(cell);
    setHigh(clk);
    delayMicroseconds(cell/2);
  }

  // read parity bit
  if (digitalRead(dat) != p) {
    // handle parity bit
  }
  delayMicroseconds(cell/2);
  setLow(clk);    
  delayMicroseconds(cell);
  setHigh(clk);
  delayMicroseconds(cell/2);
  
  // send 'ack' bit
  setLow(dat);
  delayMicroseconds(cell/2);
  setLow(clk);    
  delayMicroseconds(cell);
  setHigh(clk);
  setHigh(dat);

  return data;
}

void PS2Emu::hostWrite(unsigned char data) {
  unsigned char p = 0x01;

  // inhibit device transmission
  setLow(clk);
  delayMicroseconds(110);
  // send request to communicate with device
  setLow(dat);
  setHigh(clk);

  // send data
  for (int i=0; i<8 ;i++) {
    while (digitalRead(clk) == HIGH);
    if (data & (1 << i)) {
      setHigh(dat);
      p = p ^ 1;
    } else {
      setLow(dat);
    }
    while (digitalRead(clk) == LOW);
  }
  
  // send parity bit
  while (digitalRead(clk) == HIGH);
  if (p) {
    setHigh(dat);
  } else {
    setLow(dat);
  }
  while (digitalRead(clk) == LOW);

  // send stop bit
  while (digitalRead(clk) == HIGH);
  setHigh(dat);
  while (digitalRead(clk) == LOW);

  // eat the 'ack' bit
  while (digitalRead(clk) == HIGH);
  while (digitalRead(clk) == LOW);

}

void PS2Emu::devWrite(unsigned char data) {
  unsigned char p = 0x01;;

  // set start bit
  setLow(dat);
  delayMicroseconds(cell/2);
  setLow(clk);
  delayMicroseconds(cell);
  setHigh(clk);
  delayMicroseconds(cell/2);

  // set data bits
  for (int i=0; i<8; i++) {
    if (data & (1 << i)) {
      setHigh(dat);
      p = p ^ 1;
    } else {
      setLow(dat);
    }
    delayMicroseconds(cell/2);
    setLow(clk);
    delayMicroseconds(cell);
    setHigh(clk);
    delayMicroseconds(cell/2);
  }

  // set parity bit
  if (p) {
    setHigh(dat);
  } else {
    setLow(dat);
  }
  delayMicroseconds(cell/2);
  setLow(clk);
  delayMicroseconds(cell);
  setHigh(clk);
  delayMicroseconds(cell/2);

  // stet stop bit
  setHigh(dat);
  delayMicroseconds(cell/2);
  setLow(clk);
  delayMicroseconds(cell);
  setHigh(clk);
  delayMicroseconds(cell/2);
}

// FIXME this doesn't work
bool PS2Emu::devListen() {
  if (digitalRead(clk) == HIGH)
    return false;

  int c_stop = micros();
  int c_start = micros();

  while (digitalRead(clk) == LOW) {
    c_stop = micros();
    if (c_stop - c_start >= 100)
      return true;
  }

  return false;
}

Monday, July 15, 2013

PowerMac G5 PC Build

I come from academia where grants come and grants go and it's every professor's prerogative to burn through this year's NSF money before the next round of proposals are due. As such, the e-waste dumpster behind my building is always brimming with goodies from rack-mount servers to IBM extension cards. The other day I happened upon a pile of early 2000's Apple desktops, buried under which were these two treasures:


For all my griping about Apple, I love these machines from the first generation of brushed-aluminum-and-glass. They're so fantastically over-engineered it's more fun to take them apart than it is to actually do anything with them. Two heatsinks the size of von Clausevitz texts cool the twin PowerPC processors (which put out a whopping 1.6 GHz each) while nine gray plastic fans shuffle air around the 4-tiered case. While I'm keeping one of the G5's intact to use as a network box (pretty much all it's good for), I've decided to turn the other into an actual, useable PC. I'm going to chronicle my adventures with the PowerMac G5 in the rest of this post.

The first step is to completely gut the G5 and figure out where I stand in terms of parts. One of my goals is to maintain the exterior look of the case. The pictures below give a sense of the layout. The fans can all be re-used with a little rewiring, as can the front panel. Otherwise I think the only pieces I keep will be structural.

The ridiculous number of parts in the case.

The full tear down took hours with hundreds of tiny little torx screws that like to hide behind all of the weird structural features of the case.




/*===~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~===*/

Okay, parts have been ordered! The breakdown:

$114.99 - ASRock Z87M Pro4 (uATX motherboard)
$139.99 - G.SKILL Ripjaws X series 2X8GB (DDR3-1600, 9-9-9-24-2N)
$69.99  - Seagate Barracuda 1TB 7200RPM 64MB cache SATA-III

Total (with tax and shipping): $696.93

The toughest decision was the motherboard. I ended up going with the uATX form factor to avoid having to cut into the top tray. I like the G5's HDD rack/fan setup so I wanted to preserve it as much as possible. My only concern is the uATX I/O panel which is shorter than the G5's so I'm not sure how I'm going to adjust for that. Everything should get here Monday and I can begin the fun part of this project (more fun than dropping $700 on computer parts at least...)

/*===~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~===*/

The build


So my first step was to situate the motherboard. Everything else revolved around this. I chose to use the existing standoffs rather than a motherboard tray. To pop them out, I wrapped them in cloth (they are made of soft aluminum) and pried them off with pliers. The standoffs in the case come in two heights and I used the shorter ones for the motherboard to keep the I/O panel out of the way of the rear fans.


I aligned the motherboard to the PCI slots using an old networking card:


I secured the standoffs to the rear panel with JB weld. First I lay down some painter's tape to contain the epoxy. I scored the mounting surface with a screw driver to give the epoxy something to hold on to. Then I put the motherboard in place and wiggled it around a bit to get really get the epoxy around the standoffs. Finally I put some heavy books on the motherboard and let it cure for about 20 hours.



While the JB weld dried, I began work on the power supply. I wanted to use the original outlet so the first thing I did was slice open a standard AC power cord and solder it to the G5 outlet.


I like the plastic cover so I made sure to place the power supply far enough back in the case that it wouldn't get in the way of the strange depression in the cover (which I think is supposed to act as a venturi for the CPU fans). With the position measured I dremeled the shape of the PSU out of the top of the case. I made a space to reach the power switch as well as extra room in the back for the modular connections.


Probably the most time consuming part of this project was re-wiring all of the original G5 fans. I used three 'sets' of fans and will describe them in turn. For all three, I used these handy PWM splitter cables that I re-wired as needed:


The pins in the connectors can be removed by reaching into a slot in the front right above the pin and pushing down on a small metal tab. Remember to push the tab back up before reinserting the pin or it won't stay.


The first (and simplest) were the twin power supply fans. These aren't terribly important but I like the way they look so I left them in. The connections are very simple. One simply has a high line (red) and a ground (black), the other also has a tachometer wire (yellow). These run really loudly at 12V so rather than bother with a control (and since these don't really cool anything) I simply tied the high line to 5V (red wire in Molex connector) and left it at that.


The real fun began with the rear fan. Every fan in the G5 (except the PSU fans) is controlled individually leading to weird 4, 5, 6, and 8 pin connectors and all of the wires are black and unmarked. I used these pinouts and did not find any errors. To read them, hold the connector wires-down with the metal release tabs facing you:


To run the rear fan with a 4-pin connector, I soldered the control lines to the same pin and did away with one of the tachometer pins. I ended up with


Next up were the front CPU fans. I decided to use the one fan from this case (which was a single CPU machine) as well as the two fans from the dual CPU machine that I scavenged. I pulled the same trick with these fans, combining the control lines so that I could use the 4 pin connectors.


Since my motherboard was DOA and I had some time to kill while waiting for a replacement, I got a little creative with the fan placement. The first step was to pop the fans out of the the plastic framing, being careful not to tear the little silicon cones that hold them in.


Conveniently, the front CPU fans are exactly the depth of a 2X4 (~1.75") and so a couple of framing braces, drilled to mount the holes on the fan worked perfectly.


I used the dremel to make some aluminum bracing and held it all together with some #8x32x2" machine screws.


Lastly, I JB welded the framing braces to the rear panel and CPU cover. I think it adds just the right amount of pizzazz to the build.  


The hard drive bay fans were the last hurdles to getting the case nice and ventilated. I'm not sure why, but for whatever reason, the control of these fans seems to be only on or off. I was unable to change the speed by varying the control voltage so I resorted to simply tying the high line to 5V (like the PSU fans). Otherwise these were fairly straight forward. The reason for the 5-pin connector on the blower is also something of a mystery.


And the completed HDD bay with SATA cables, power and fans:


The only change I had to make to the exterior of the case was cutting the I/O panel out. As much as I would have like to rewire everything into the original ports, it made the prospect of ever switching motherboards pretty daunting. Instead, I simply cut the whole I/O panel out and will hopefully get around to crafting a nice faceplate out of sheet aluminum in the future. For now, it's just open to the world.


With all of the fans in place and the wiring all sorted out, I was ready to put the motherboard back in.
 
I installed the processor and memory and carefully lowered the motherboard in place.





The final step to a working machine was wiring the front panel to the motherboard. I used the following diagram for the front panel connector:



The only difficult part here is that the power button is connected to the firewire ground pin (top left of the diagram) and only to this ground. So three wires sufficed for the front panel: motherboard +PWR to pin 13 in the above diagram; motherboard +PLED to pin 14; and finally motherboard GND to firewire ground (not the shielded gnd pin) - pin 2 above.


And with that I was able to fire it right up. There are still a few issues that I need to deal with. The first is that my motherboard cannot seem to control the G5's fans. Apple being Apple, they are likely controlled by some very specifically shaped pulse known only to the engineers who developed the fans and who have now been permanently stationed on Eros (Ender's Game anyone?) For now the control is tied to 5V which seems to keep them at a reasonable speed. I would like to wire the front USB port and front headphone jack but I need to go out an buy connectors to fit the corresponding motherboard headers. Otherwise I'm rather pleased with the build. Kubuntu 13.04 installed with absolutely no drama and within a few hours this has become my new work machine. All in all, very pleased with this project.