Friday, December 27, 2013

Amtrak real-time train data

Recently the US passenger rail operator Amtrak announced a nationwide real-time map of it's services.

That's great...until you try it.

The United States is big, even though the number of trains is fairly small. The number of Google map tiles is big. The polygons that show the routes are enormous. The nationwide train data is big.

That's a lot of data to load --and so slow to display-- if all I really want is realtime data on a single train (is my train late?).
 
After looking at the GETs emitted by all the (slow, unnecessary) javascript on that site, I figured out how to bypass it and pull just the table of current train status. It's 400kb, but it's a lot less than the over 1MB to go through the website and map....

https://www.googleapis.com/mapsengine/v1/tables/01382379791355219452-08584582962951999356/
features?version=published&key=AIzaSyCVFeFQrtk-ywrUE0pEcvlwgCqS6TJcOW4&maxResults=250&
callback=jQuery19106722136430546803_1388112103417&dataType=jsonp&
jsonpCallback=retrieveTrainsData&contentType=application%2Fjson&_=1388112103419


The only fields required are 'version' and 'key' (which is different from the site cookie):

https://www.googleapis.com/mapsengine/v1/tables/01382379791355219452-08584582962951999356/
features?version=published&key=AIzaSyCVFeFQrtk-ywrUE0pEcvlwgCqS6TJcOW4

So we can easily download the current status table using wget:

$ wget -O train_status https://www.googleapis.com/mapsengine/v1/tables/
01382379791355219452-08584582962951999356/features?version=published\&
key=AIzaSyCVFeFQrtk-ywrUE0pEcvlwgCqS6TJcOW4\&jsonpCallback=retrieveTrainsData


Since parsing 400kb of JSON using shell script will be awkward and annoying, let's change over to Python3:

#/usr/bin/python3
import httplib2
url   = \
"""https://www.googleapis.com/mapsengine/v1/table
/01382379791355219452-08584582962951999356/features?version=published&
key=AIzaSyCVFeFQrtk-ywrUE0pEcvlwgCqS6TJcOW4&jsonpCallback=retrieveTrainsData"""

h = httplib2.Http(None)
resp, content = h.request(url, "GET")

Let's tell Python that the content is a JSON string, and iterate through the list of trains looking for train number 7. That's a handy train because each run takes over 24 hours - it will always have a status.

import json
all_trains = json.loads(content.decode("UTF-8"))
for train in all_trains["features"]:
   if train["properties"]["TrainNum"] == "7":
       do something

After locating the train we care about, here is data about it:

latitude     = train["geometry"]["coordinates"][0]
longitude    = train["geometry"]["coordinates"][1]
speed        = train["properties"]["Velocity"]
report_time  = train["properties"]["LastValTS"]
next_station = train["properties"]["EventCode"]


Let's find a station (FAR) along the route. This is a little tougher, because Python's JSON module reads each Station as a string instead of a dict. We need to identify Station lines, and feed those through the JSON parser (again).

for prop in train["properties"].keys():
          if "Station" in prop:
              sta = json.loads(train["properties"][prop])
              if sta["code"] == "FAR":
                  # Schedule data
                  station_time_zone        = sta['tz']
                  scheduled_arrival_time   = sta['scharr'] # Only if a long stop
                  scheduled_departure_time = sta['schdep'] # All stations

                  # Past stations
                  actual_arrival_time      = sta['postarr']
                  actual_departure_time    = sta['postdep']
                  past_station_status      = sta['postcmnt']

                  # Future stations
                  estimated_arrival_time   = sta['estarr']
                  future_station_status    = sta['estarrcmnt']

The final Python script for Train 7 at FAR is:

#/usr/bin/python3

import httplib2
import json

url = "https://www.googleapis.com/mapsengine/v1/tables/01382379791355219452-08584582962951999356/features?version=published&key=AIzaSyCVFeFQrtk-ywrUE0pEcvlwgCqS6TJcOW4&jsonpCallback=retrieveTrainsData"
h = httplib2.Http(None)
resp, content = h.request(url, "GET")

all_trains = json.loads(content.decode("UTF-8"))
for train in all_trains["features"]:
   if train["properties"]["TrainNum"] == "7":
      for prop in train["properties"].keys():
          if "Station" in prop:
              sta = json.loads(train["properties"][prop])
              if sta["code"] == "FAR":
                  print('Train number {}'.format(train["properties"]["TrainNum"]))
                  if 'schdep' in sta.keys():
                      print('Scheduled: {}'.format(sta['schdep']))
                  if 'postcmnt' in sta.keys():
                      print("Status: {}".format(sta['postcmnt']))
                  if 'estarrcmnt' in sta.keys():
                      print("Status: {}".format(sta['estarrcmnt']))

And the result looks rather like:

$ python3 Current\ train\ status.py 

Train number 7
Scheduled: 12/26/2013 03:35:00
Status: 1 HR 4 MI LATE

Train number 7
Scheduled: 12/27/2013 03:35:00




Two more items.

First, the Python script can be tweaked to show *all* trains for a station. Just remove the train number filter. To be useful, you might add some time comprehension, since some scheduled times can be a day in the future or past.

Second, the station information table has a similar static page: https://www.googleapis.com/mapsengine/v1/tables/01382379791355219452-17620014524089761219/features?&version=published&maxResults=1000&key=AIzaSyCVFeFQrtk-ywrUE0pEcvlwgCqS6TJcOW4&callback=jQuery19106722136430546803_1388112103417&dataType=jsonp&jsonpCallback=retrieveStationData&contentType=application%2Fjson&_=13881121034

As with the train status JSON application, many of the fields are dummies. This shorter URL works, too: https://www.googleapis.com/mapsengine/v1/tables/01382379791355219452-17620014524089761219/features?&version=published&key=AIzaSyCVFeFQrtk-ywrUE0pEcvlwgCqS6TJcOW4&callback=jQuery19106722136430546803_1388112103417




Sunday, December 22, 2013

Python http.server and upstart-socket-bridge

In a previous post, I showed how to make the Python 3 socketserver.TCPServer class compatible with upstart-socket-bridge by overriding the server_bind() and server_activate() methods.

Python's http.server module builds on socketserver. Let's see if we can similarly make http.server compatible with upstart-socket-bridge.

About http.server

The http.server module is intended to be a simple way to create webservers. Most of the module is devoted to classes that handle incoming and outgoing data. Only one class, HTTPServer, handles the networking stuff.




Example 1

Here is the first example http.server.SimpleHTTPRequestHandler in the Python documentation:

import http.server
import socketserver

PORT = 8000

Handler = http.server.SimpleHTTPRequestHandler

httpd = socketserver.TCPServer(("", PORT), Handler)

print("serving at port", PORT)
httpd.serve_forever()

When you run this code, and point a web browser to port 8000, the code serves the current working directory, with links to files and subdirectories.


Convert Example 1 to work with upstart-socket-bridge

The example server is only seven lines.

Line 3, the PORT, is no longer needed. Upstart will define the port.

Line 5, the socketserver.TCPServer line, will be the biggest change. We need to define a new class based on TCPServer, and override two methods. The is exactly what we did in the previous post.

Line 6, the print statement, can be deleted. When the job is started by Upstart, there is no display - nowhere to print to. An important side effect of starting the job using Upstart is that the Present Working Directory is / (root), unless you specify otherwise in the /etc/init config file that starts the job.

Because of the change to Line 5, the final result is 6 more lines...not bad, and now it works with upstart-socket-bridge.

import http.server
import socketserver

class MyServer(socketserver.TCPServer):
    def server_bind(self):
        """ Replace the socket FD with the Upstart-provided FD"""
        fd = int(os.environ["UPSTART_FDS"])
        self.socket = socket.fromfd(fd, socket.AF_INET, socket.SOCK_STREAM)

    def server_activate(self):
        pass            # Upstart takes care of listen()

Handler = http.server.SimpleHTTPRequestHandler
server = MyServer(None, Handler)
server.serve_forever()

As always, the script is triggered by an appropriate Upstart job.

Test the script, in this case, by pointing a web browser at the port specified in the Upstart job.




Example 2


This example is also from the Python http.server documentation:

def run(server_class=HTTPServer, handler_class=BaseHTTPRequestHandler):
    server_address = ('', 8000)
    httpd = server_class(server_address, handler_class)
    httpd.serve_forever()

It's not a standalone example, but instead an abstract example of how to use the module in a larger application.

Let's reformat it into a working example:

import http.server
server_address = ('', 8000)
handler_class=http.server.BaseHTTPRequestHandler
httpd = http.server.HTTPServer(server_address, handler_class)
httpd.serve_forever()

Now the server runs...sort of. It gives us a 501 error message. Let's add a class to the handler that reads the path and gives a real, valid response. (Source)

import http.server

class MyHandler(http.server.BaseHTTPRequestHandler):
    def do_HEAD(self):
        self.send_response(200)
        self.send_header("Content-type", "text/html")
        self.end_headers()
    def do_GET(self):
        self.send_response(200)
        self.send_header("Content-type", "text/html")
        self.end_headers()
        print(self.wfile)
        content = ["<html><head><title>The Title</title></head>",
                   "<body>This is a test.<br />",
                   "You accessed path: ", self.path, "<br />",
                   "</body></html>"]
        self.wfile.write("".join(content).encode("UTF-8"))


server_address = ('', 8000)
handler_class=MyHandler
httpd = http.server.HTTPServer(server_address, handler_class)
httpd.serve_forever()

Okay, now this is a good working example of a working http.server. It really shows how http.server hides all the complexity of networking on one simple .server line and focuses all your effort on content in the handler. It's clear how you would parse the URL input using self.path. It's clear how you can create and send content using self.wfile.

Convert Example #2 to work with upstart-socket-bridge

There are two classes at work: http.server.BaseHTTPRequestHandler handles content doesn't care about the networking. http.server.HTTPServer handles netwokring and doesn't care about content.

Good news: HTTPServer is based on socketserver.TCPServer, which we already know how to patch!

The changes I made:

1) I want it to launch at each connection, exchange data once, and then terminate. Each connection will launch a separate instance. We no longer want the service to serve_forever().

httpd.serve_forever()    # Old
httpd.handle_request()   # New



2) Let's make it compatible with all three inits: sysvinit (deamon), Upstart, and systemd.

import os

if __name__ == "__main__":
    if "UPSTART_FDS" in os.environ.keys() \   # Upstart
    or "LISTEN_FDS" in os.environ.keys():     # systemd
        httpd = MyServer(None, MyHandler)     # Need a custom Server class
        httpd.handle_request()                # Run once
    else:                                     # sysvinit
        server_address = ('', 8000)
        httpd = http.server.HTTPServer(server_address, MyHandler)
        httpd.serve_forever()                 # Run forever



3) Add a custom handler that overrides server_bind() and server_activate() in both the socketserver and http.server.HTTPServer modules. This is the secret sauce that makes Upstart compatibility work:

import http.server, socketserver, socket, os

class MyServer(http.server.HTTPServer):
    def server_bind(self):
        # Get the File Descriptor from an Upstart-created environment variable
        if "UPSTART_FDS" in os.environ:
            fd = int(os.environ["UPSTART_FDS"])      # Upstart
        else:
            fd = int(os.environ["LISTEN_FDS"])       # Systemd
        self.socket = socket.fromfd(fd, socket.AF_INET, socket.SOCK_STREAM)

        # Only http.server.CGIHTTPRequestHandler uses these
        host, port = self.socket.getsockname()[:2]
        self.server_name = socket.getfqdn(host)
        self.server_port = port



4) Add the Upstart job to monitor the port:

# /etc/init/socket-test.py
description "upstart-socket-bridge test"
start on socket PROTO=inet PORT=8000 ADDR=127.0.0.1
setuid your_username               # Does not need to run as root
exec /usr/bin/python3 /tmp/socket-server.py



And the final product looks like:

#!/usr/bin/python3

import http.server, socketserver, socket, os

class MyServer(http.server.HTTPServer):
    """
    This class overrides two methods in the socketserver module:
        socketserver __init__ uses both server_bind() and server_activate()
    This class overrides one method in the http.server.HTTPServer class:
        HTTPServer uses both socketserver __init__ and it's own custom
        server_bind
    These overrides makes it compatible with Upstart and systemd socket-bridges
    Warning: It won't bind() or listen() to a socket anymore
    """
    def server_bind(self):
        """
        Get the File Descriptor from an Upstart-created environment variable
        instead of binding or listening to a socket.
        """
        if "UPSTART_FDS" in os.environ:
            fd = int(os.environ["UPSTART_FDS"])
        else:
            fd = int(os.environ["LISTEN_FDS"])
        self.socket = socket.fromfd(fd, socket.AF_INET, socket.SOCK_STREAM)

        # From http.server:
        # http.server.CGIHTTPRequestHandler uses these.
        # Other handler classes don't use these.
        host, port = self.socket.getsockname()[:2]
        self.server_name = socket.getfqdn(host)
        self.server_port = port


    def server_activate(self):
        """
        This socketserver method sends listen(), so it needs to be overridden
        """
        pass


class MyHandler(http.server.BaseHTTPRequestHandler):
    """
    A very simple custom handler.
    It merely reads the URL and responds with the path.
    This shows how you read data from a GET, and send a response. 
    """
    def do_HEAD(self):
        self.send_response(200)
        self.send_header("Content-type", "text/html")
        self.end_headers()
    def do_GET(self):
        self.send_response(200)
        self.send_header("Content-type", "text/html")
        self.end_headers()
        print(self.wfile)
        content = ["The Title",
                   "This is a test.
",
                   "You accessed path: ", self.path, "
",
                   ""]
        self.wfile.write("".join(content).encode("UTF-8"))


if __name__ == "__main__":
    if "UPSTART_FDS" in os.environ.keys() \     # Upstart
    or "LISTEN_FDS" in os.environ.keys():       # systemd
        httpd = MyServer(None, MyHandler)       # Use fd to get connection
        httpd.handle_request()                  # Handle once, then terminate
    else:
        server_address = ('', 8000)             # sysvinit, classic bind()
        httpd = http.server.HTTPServer(server_address, MyHandler)
        httpd.serve_forever()


Test the service:

  • The Python 3 script and Upstart job both use Port 8000.
  • Save the Python 3 script. Make it executable.
  • Save the Upstart job. Change the Upstart job to Port 8001. Make sure it points to the Python 3 script.

Webserver daemon using sysvinit - run forever:
  • Run the Python 3 script.
  • Start a web browser, and point it to http://localhost:8000/test/string
  • The browser should show a response
  • Kill the Python 3 script

Web service using upstart - run once:

  • Don't start the Python 3 script. Upstart will do it for you.
  • Start a web browser, and point it to http://localhost:8001/test/string
  • The browser should show a response.








Tuesday, December 10, 2013

Python socketserver and upstart-socket-bridge

In a previous post, I described how to use Upstart to monitor a port, trigger a Python script upon connection, and then let the Python application send and receive data over that port.

In a server application, Upstart can take over the bind() and listen() elements of the server daemon, sometimes eliminating the need for a daemon at all.

Here is an example of a Python socketserver, and how it changes when using Upstart.

Some complexity is reduced: The handler is simpler, it no longer needs to fork() or serve_forever(). Each connection triggers the application, so the application need only handle a single read()/write() cycle, then terminate.

Some complexity is increased: We need to override two methods of a socketserver class to make it upstart-socket-bridge compatible.




Here is the original socketserver.TCPServer serve_forever example, from the Python documentation:

import socketserver

class MyTCPHandler(socketserver.BaseRequestHandler):
    """
    The RequestHandler class for our server.

    It is instantiated once per connection to the server, and must
    override the handle() method to implement communication to the
    client.
    """

    def handle(self):
        # self.request is the TCP socket connected to the client
        self.data = self.request.recv(1024).strip()
        print("{} wrote:".format(self.client_address[0]))
        print(self.data)
        # just send back the same data, but upper-cased
        self.request.sendall(self.data.upper())

if __name__ == "__main__":
    HOST, PORT = "localhost", 9999

    # Create the server, binding to localhost on port 9999
    server = socketserver.TCPServer((HOST, PORT), MyTCPHandler)

    # Activate the server; this will keep running until you
    # interrupt the program with Ctrl-C
    server.serve_forever()


It's easy to test. Let's save the file as /tmp/socket-server-original.py

$ python3 /tmp/socket-server-original.py     # On Terminal #1

$ echo "Test String" | nc localhost 9999     # On Terminal #2
TEST STRING

The example works.




Now, let's adapt it for upstart-socket-bridge.

It takes a bit of reading through the socketserver source code to figure it out:

TCPServer's __init__ calls good old bind() and listen() using it's own methods server_bind() and server_activate(), respectively.

Both of those methods can be overridden. Better yet, they use socket module calls (socket.bind() and socket.listen()) that we already know how to replace with socket.fromfd().

So we can get TCPServer to work by overriding the two methods like this:

import os, socket

class MyServer(socketserver.TCPServer):
    """
    This class overrides two method in socketserver.TCPServer 
    and makes it Upstart-compatible
    """
    def server_bind(self):
        """ Replace the socket-created file descriptor (fd) with the Upstart-provided fd"""
        fd = int(os.environ["UPSTART_FDS"]
        self.socket = socket.fromfd(fd, socket.AF_INET, socket.SOCK_STREAM)
def server_activate(self): """This means listen(), but we don't want it do do anything""" pass


Changing the TCPServer class slightly changes the way way we activate.
We must call the new class instead of TCPServer, and we no longer need the address since bind() no longer does anything:

    server = socketserver.TCPServer((HOST, PORT), MyTCPHandler)  # OLD
    server = MyServer(None, MyTCPHandler)                        # NEW


The code that handles the request, generates the response, and sends the response has two changes. Printing on the terminal won't work, since Upstart doesn't use a terminal to start the process. So we'll delete the printing lines:

        print("{} wrote:".format(self.client_address[0]))
        print(self.data)





Finally, one more change. The script can quit after the response has been sent:

    server.serve_forever()      # OLD
    server.handle_request()     # NEW





When we put it all together, here is the new version that works with upstart-socket-bridge:

import socketserver, os, socket

class MyServer(socketserver.TCPServer):
    """
    This class overrides two method in socketserver.TCPServer 
    and makes it Upstart-compatible
    """
    def server_bind(self):
        """ Replace the socket FD with the Upstart-provided FD"""
        fd = int(os.environ["UPSTART_FDS"])
        self.socket = socket.fromfd(fd, socket.AF_INET, socket.SOCK_STREAM)

    def server_activate(self):
        pass            # Upstart takes care of listen()


class MyTCPHandler(socketserver.BaseRequestHandler):
    """
    The RequestHandler class for our server.

    It is instantiated once per connection to the server, and must
    override the handle() method to implement communication to the
    client.
    """

    def handle(self):
        # self.request is the TCP socket connected to the client
        self.data = self.request.recv(1024).strip()

        # just send back the same data, but upper-cased
        self.request.sendall(self.data.upper())

if __name__ == "__main__":
    # Create the server using the file descriptor from Upstart
    server = MyServer(None, MyTCPHandler)

    # Run once, then terminate
    server.handle_request()


Let's save the file as /tmp/socket-server-new.py
We'll need to adapt (or create) /etc/init/socket-test.conf Upstart job file.

description "upstart-socket-bridge test"
start on socket PROTO=inet PORT=34567 ADDR=127.0.0.1
setuid myusername   # Use *your* user name. Doesn't need to be root
exec /usr/bin/python3 /tmp/socket-server-new.py

And let's give it a try on the terminals:

$ python3 /tmp/socket-server-new.py           # On Terminal #1

$ echo "Test String" | nc localhost 34567     # On Terminal #2
TEST STRING

Overriding socketserver for upstart compatibility works.




After this test, cleanup is easy

$ sudo rm /etc/init/socket-test.conf
$ rm /tmp/socket-server-original.py /tmp/socket-server-new.py

Monday, December 2, 2013

upstart-socket-bridge

Upstart-socket-bridge is a lot like xinetd. They replace the need for some daemons by monitoring a port, and then launching the desired application when an inbound connection is detected. U-s-b is part of a family of Upstart services that replace many daemon monitoring and listening functions and hooks.

Unlike xinetd, services need to be customized (patched) to run with upstart-socket-bridge.

Documentation is quite sparse. Hence this blog post. That's not intended to criticize; it's really hard to write "good" documentation when you don't know the use cases or the experience level of the user. If you have experience with writing sockets in C, and understand what a file descriptor is and how to use one,  then the documentation is just fine. I didn't before I began this odyssey.




How do I make it work?

Here are three simple examples of how it works.
One uses shell script.
One uses Python3.
One uses C.



Hello, World! with shell script


The script that gets triggered by the port action. The port is just a trigger, no data gets exchanged on the port.

1) Let's create a shell script called test-script. This script merely prints out the Upstart-related environment variables into a file.

#!/bin/sh
outfile=/tmp/outfile
date > $outfile            # Timestamp
printenv | grep UPSTART >> $outfile
exit 0


2)  Create an Upstart .conf, let's call it /etc/init/socket-test.conf

description "upstart-socket-bridge test"
start on socket PROTO=inet PORT=34567 ADDR=127.0.0.1  # Port 34567
setuid exampleuser                                    # Run as exampleuser, not root
exec /bin/sh /tmp/test-script                         # Launch the service


3)  Let's run it. Connect to the port using netcat.

$ nc localhost 34567
^C       # End the process using CTRL+C


4)  Look at the result. Hey, look, it's all the environment variables we need!

$ cat /tmp/outfile


5)  Clean up:

$sudo rm /etc/init/socket-test.conf           # Disconnect the launch trigger
$rm /tmp/test-script /tmp/outfile             # Delete the test service





"Hello, World" service in Python 3

(UPDATED: Thanks to Dmitrijs Ledkovs for getting this to work!)

It's a simple echo server - the Python version of the C service below. It requires two files, the application and the Upstart .conf. It demonstrates how a service reads uses the port connection for a trigger and exchanging data.


1) Let's create the Python 3 file. Let's call it test-service.py

#!/usr/bin/python3
import os, socket

# Create the socket file descriptor from the the env var
sock_fd = socket.fromfd(int(os.environ["UPSTART_FDS"]),
                        socket.AF_INET, socket.SOCK_STREAM)

# Accept the connection, create a connection file descriptor
conn, addr = sock_fd.accept()

# Read
message = conn.recv(1024).decode('UTF-8')

# Manipulate data
reply = ("I got your message: " + message)

# Write
conn.send(reply.encode('UTF-8'))

# Finish
conn.close()



2)  Create an Upstart .conf, let's call it /etc/init/socket-test.conf

description "upstart-socket-bridge test"
start on socket PROTO=inet PORT=34567 ADDR=127.0.0.1  # Port 34567
setuid exampleuser                                    # Run as exampleuser, not root
exec /usr/bin/python3 /tmp/test-service.py            # Launch the service


3) Let's run it. Connect to the port using netcat, and then type in a string.

$ nc localhost 34567
Hello, World!                       # You type this in. Server read()s it.
I got your message: Hello, World!   # Server response.  Server write()s it.


4) Cleanup is simple. Simply delete the two files.

$ sudo rm /etc/init/socket-test.conf         # Disconnect the bridge
$ rm /tmp/test-service.py                    # Delete the test service







"Hello, World!" service in C


It's a simple echo server - the C version of the Python service above. It requires two files, the application and the Upstart .conf. It demonstrates how a service reads uses the port connection for a trigger and exchanging data.

1)  Let's create a C file. Let's call it test-service.c

#include <stdlib.h>
#include <netinet/in.h>
#include <string.h>

int main()
{
    /* Read the UPSTART_FDS env var to get the socket fd */
    char *name = "UPSTART_FDS";
    char *env = getenv (name);       // Read the environment variable
    int sock_fd = atoi(env);         // Socket file descriptor

    /* Don't need to do any of these normal socket tasks! Hooray!
    / int port_num;           
    / int sock_fd = socket(AF_INET, SOCK_STREAM, 0);  
    / memset((char *) &serv_addr, 0, sizeof(serv_addr));
    / serv_addr.sin_family = AF_INET;
    / serv_addr.sin_addr.s_addr = INADDR_ANY;
    / serv_addr.sin_port = htons(port_num);
    / struct sockaddr_in serv_addr
    / bind(sock_fd, (struct sockaddr *) &serv_addr, sizeof(serv_addr));
    / listen(sock_fd, 5)                                                 
    */

    /* Accept() the connection. Returns the second fd: 'conn_fd' */
    struct sockaddr_in cli_addr;   // Requires netinet/in.h
    int clilen = sizeof(cli_addr);
    int conn_fd = accept(sock_fd, (struct sockaddr *) &cli_addr, &clilen);

    /* Service is active. Read-from and write-to the connection fd */
    char response[276] = "I got your message: ";
    char buffer[256];
    memset((char *) &buffer, 0, sizeof(buffer));  
    read(conn_fd, buffer, 256);                   // Read from conn_fd
    strcat(response, buffer);                     
    write(conn_fd, response, strlen(response));   // Write to conn_fd

    /* Close the connection fd. Socket fd can be reused */
    close(conn_fd);
    return 0;
}

2)  Compile it using gcc, and output the compiled application as an executable called test-service. I put mine in /tmp to make cleanup easier. If you're familiar with gcc, the important element is that there are no flags and no libraries:

gcc -o test-service test-service.c


3)  Create an Upstart .conf, let's call it /etc/init/socket-test.conf

description "upstart-socket-bridge test"
start on socket PROTO=inet PORT=34567 ADDR=127.0.0.1  # Port 34567
setuid exampleuser                                    # Run as exampleuser, not root
exec /tmp/test-service                                # Launch the service


4) Let's run it. Connect to the port using netcat, and then type in a string.

$ nc localhost 34567
Hello, World!                       # You type this in. Server read()s it.
I got your message: Hello, World!   # Server response.  Server write()s it.


5) Cleanup is simple. Simply delete the three files.

$ sudo rm /etc/init/socket-test.conf         # Disconnect the bridge
$ rm /tmp/test-service.c /tmp/test/service   # Delete the test service



How does it work?

Here is the oversimplified explanation. Each stream of data whizzing round inside your system is tracked by the kernel. That tracking, sort of like an index or a pointer, is called a file descriptor (fd). A few fds are reserved (0=stdin, 1=stdout, 2=stderr) and you run into these in shell scripting or cron jobs.

A pipe or port or socket is just a way to tell the kernel that a stream of data output from Application A should be input to Application B. Let's look at it another way, and add that fd definition: An fd identifies a data stream output from A and input to B. The pipe/socket/port is a way to express how you want the fd set up.

Now the gritty stuff: A socket/port can have a single server and multiple clients attached. The server bind()s the port, and listen()s on the port for connections, and accept()s each connection from a client. Each connection to a client gets issues another file descriptor.

That's two file descriptors: The first defines the port/socket in general, and the second defines the specific connection between one client and the server.

Upstart tells your server application (since it's not actually serving, let's just call it the "service") the first file descriptor.
  • Your service doesn't start a daemon at boot.
  • Your service doesn't bind() to the socket/port. Upstart does that.
  • Your service doesn't listen() on the socket/port. Upstart does that.
  • Your service doesn't fork() for each connection. Upstart launches an instance of your service for each connection. Your service can terminate when the connection ends...if you want it to.
  • Your service does accept() the connection from a client, communicate using the resulting file descriptor, and end when the connection close()s.

Let's try it with the example above:
  1. Upstart and Service are running on Server. Client is running somewhere else - maybe it's also running on Server, maybe it's out on the network somewhere.
  2. The file /etc/init/socket-test.conf tells Upstart to monitor port #34567 on behalf of test-service application. As currently written, it will begin monitoring at boot and stop monitoring at shutdown.
  3. When Client --like netcat-- connect()s to port #34567, Upstart launches test-service application with a couple extra environment variables.
  4. test-service reads the environment variables, including the file descriptor (fd).
  5. test-service accept()s the connection on the file descriptor. This creates a second fd that Service can use to communicate.
  6. When Client and test-service are done communicating, they close() the connection fd.
  7. test-service can end. Upstart will restart it next time an inbound connection is received.




How do I make it work with a service I didn't write? (like my favorite Game Server or Media Server or Backup Server)

Maybe it will work, maybe it won't. There are a couple issues to consider. I don't see an easy, non-coding solution because we're talking about changing the nature of these services.
  • Change from always-on to sometimes-on.
  • Change to (save and) quit when the connection is done instead of listen()ing for another connection. I don't see any upstart trigger for a socket connection ending.
  • Might make some service code simpler. No longer need to fork() or bind().
  • Not portable to non-Upstart systems, so the daemon code remains. Adds a bit to code complexity and a new testing case.
  • A different trigger (hardware, file change, etc) might be a better trigger than a connection to a socket. Upstart has bridges for those, too.

Sunday, December 1, 2013

Handy Pipe and Socket Reference

I'm not going to explain sockets. There are too many good tutorials out there already.
Instead, I'm going to use sockets in a bunch of examples that build.

Ordinary Pipes
Named Pipes/Sockets
Unix Domain Sockets
TCP Sockets/Ports
upstart-socket-bridge


The Simplest


Pipes (|) and redirection (>). We already know how to do that in the shell.
These pipes connect related processes. For example, the children of a single terminal are related.

It's hard to pipe from shell to python (or the reverse) - it's possible, just not easy the first couple times you try.

Shell example:
$ echo Hello, Planet\! | sed 's/Planet/World/'
Hello, World!

Here is a good example of pipes in Python 3.




Named Pipes


Slightly more complicated (reference). These use a pipe file that synchronizes the data transfer when both processes are ready. A -> Pipe -> B. This is handy for inter-process communication that does not need to run through dbus, and saves a bit of overhead.

Names pipes are a shared file. They can have many (anonymous) writers, but only a single reader. Shell, Python, and C senders and listeners can interoperate (fairly) smoothly.

Named pipes are blocking. Start the reader first, and it will wait for the writer. Alternately, start the writer first, and it will wait for the reader. Once the writer has finished, the reader will also stop. One read() for one write(). A second read() requires a second write()...and a second write() requires a second read().

Named pipes in shell and C can be reused. Named pipes in Python cannot be reused after close() - they must be unlink() (destroyed) and remade afresh.

Conventionally, the listener is expected to create the pipe. But it's not required.

Processes that use named pipes work in the following order:
  • Create the pipe (usually done by listener)
  • Open the pipe (both sides)
  • Write the pipe
  • Writer close the pipe (allows the reader to see the data)
  • Listener read the pipe
  • Listener close the pipe
  • Delete the pipe (usually done by the listener)


Creating and Deleting a Named Pipe:



Shell:

#!/bin/sh
mkfifo /tmp/testpipe        # Create pipe
rm /tmp/testpipe            # Delete pipe

Python3:

>>> import os
>>> os.mkfifo("/tmp/testpipe")    # Create pipe
>>> os.remove("/tmp/testpipe")    # Delete pipe 

C:

# include <sys/stat.h>
int main( ){
   int result = mkfifo ("/tmp/testpipe", S_IRUSR| S_IWUSR);  \\ Create Pipe
   unlink ("/tmp/testpipe");                                 \\ Delete Pipe
   return 0;
   }


Named Pipe Listeners:

 

Shell:

#!/bin/sh
pipe=/tmp/testpipe
trap "rm -f $pipe" EXIT
while true
do
    read line <$pipe   # Read the pipe
    echo $line
done

Python3:

>>> import os
>>> pipe = open("/tmp/testpipe", "r")
>>> pipe.read()     # You won't read anything until the sender close()
'Hello, World!'     # Data from pipe
>>> pipe.close()

C:

#include <fcntl.h>
#include <stdio.h>
int main()
{
    char buf[20];
    int fd = open("/tmp/testpipe", O_RDONLY);  // Open the pipe
    read(fd, buf, 20);                         // Read the message - unblock the writing process
    printf("Received: %s\n", buf);             // Display the message
    close(fd);                                 // Close the pipe
    return 0;
}


Named Pipe Senders:


Shell:

#!/bin/sh
pipe=/tmp/testpipe
[ ! -p $pipe ] && echo "Reader not running" && exit 1
[ "$1" ] && echo "$1" >$pipe || echo "Hello from $$" >$pipe

Python3:

>>> import os
>>> pipe = open("/tmp/testpipe", "w") # Hang here awaiting the receiver to read() once
>>> pipe.write("Hello, World!")
13
>>> pipe.close()    # Receiver read() is empty until this happens

C:

#include <fcntl.h>
int main()
{
    int fd = open("/tmp/testpipe", O_WRONLY);   // Open the pipe
    write(fd, "Hi", sizeof("Hi"));              // Write "Hi" to pipe
    close(fd);                                  // Close the pipe - allow the read
    return 0;
}





Unix Domain Sockets


Unix Domain Sockets is a fancy name for a fancy pipe. Like shell pipes, Unix Sockets use a pipe file...Well it *looks* like a pipe file. Actually, data is buffered instead of written to a file. The file is simply an easy way for the processes to find each other.

Unlike shell pipes, Unix sockets are bi-directional, and multiple writers are not anonymous. In Python, sockets are single-use: After a close() they cannot be reopened, and must be deleted and remade. Pipes can only be read() when the sender close(), but sockets can be read() for every send().

Since sockets are bi-directional, it's not a sender/receiver relationship anymore. It's a client/server relationship.

Processes using a Unix Domain Socket usually work in the following order:
  • Create a file descriptor (fd) that defines the socket OR look up the fd of an existing socket.
  • Server binds to the socket
  • Server listens on the socket
  • Client connects to the socket
  • Server accepts the connection
  • Both sides read/write
  • Client closes socket (on Python, cannot be reopened - must be reconnected )
  • Server closes socket (usually during service shutdown)
  • Delete/unlink the file descriptor and socket (if it won't be reused)

Creating and Deleting a Unix Domain Socket:


Shell:

Shell creation can be automatic, for example using netcat:
$ nc -lU /tmp/sock           # Create socket, among other actions
$ rm /tmp/sock               # Delete socket

Python3:

>>> import socket, os
>>> tempsocket = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)  
>>> tempsocket.bind("/tmp/sock")  # Create socket
>>> os.remove("/tmp/sock")        # Delete socket 

C:

#include <sys/socket.h>
#include <sys/un.h>
int main() {
  char *path = "/tmp/sock";
  struct sockaddr_un addr;
  char buf[100];
  int cl,rc;
  int fd = socket(AF_UNIX, SOCK_STREAM, 0);   // Create a file descriptor
  addr.sun_family = AF_UNIX;                  // It's a Unix socket
  strcpy(addr.sun_path, path);                // Set the path
  memset(&addr, 0, sizeof(addr));         // Fill the address array with zeroes (prevent garbage)
  bind(fd, (struct sockaddr*)&addr, sizeof(addr));    // Create socket (server)
  connect(fd, (struct sockaddr*)&addr, sizeof(addr)); // Connect (client)
  unlink(path);                               // Delete socket
  return 0;
}


Unix Domain Socket Servers (Bind to socket):

The server owns (binds) and monitors the socket, and is usually responsible for creating and deleting the socket. (C example)

Shell:

$ nc -lU /tmp/sock

Python3:

>>> import socket
>>> tempsocket = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)
>>> tempsocket.bind("/tmp/sock")       # Listener is first process using socket
>>> tempsocket.listen(1)
>>> conn, addr = tempsocket.accept()   # Listener blocks here awaiting sender's connect()
>>> conn.recv(1024).decode('UTF-8')    # Listener blocks here awaiting send()
'Hello, World'                         # Try this a few times to capture each send()
>>> tempsocket.close()

C:

#include <stdio.h>
#include <sys/socket.h>
#include <sys/un.h>
int main() {
  char buf[100];
  int cl,rc;
  int fd = create_socket(socket_path)      // Creating the socket is not done here. See above
  listen(fd, 5);                // listen() is the step after bind()
  while (1) {
    cl = accept(fd, NULL, NULL);
    while ( (rc=read(cl,buf,sizeof(buf))) > 0) {
      printf("read %u bytes: %.*s\n", rc, rc, buf);
    }
    if (rc == 0) {
      printf("EOF\n");
      close(cl);
    }
  }
  return 0;
}



Unix Domain Socket Clients (Connect to socket):

Clients don't own (bind) the socket. Instead, they connect on-demand.

Shell:

$ nc -U /tmp/sock

Python3:

>>> import socket
>>> tempsocket = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)
>>> tempsocket.connect("/tmp/sock")           # Listener must be active before sender
>>> tempsocket.send(data)                     # Try this a few times
13
>>> tempsocket.close()                        # End of stream

C:

#include <sys/socket.h>
#include <sys/un.h>
#include <unistd.h>
int main() {
  struct sockaddr_un addr;
  char buf[100];
  int rc;
  int fd = create_socket(socket path)  // Creating the socket is not done here

  // First step after connect()  
  while( (rc=read(STDIN_FILENO, buf, sizeof(buf))) > 0) write(fd, buf, rc);
  return 0;
}



Quick fun with netcat:

The easiest possible bi-directional socket.
$ nc -lU /tmp/sock   # Enter in Terminal #1
$ nc -U /tmp/sock    # Enter in Terminal #2

Type anything on either terminal.
Look! The two communicate!

Here is an example of using netcat to exchange files this way (from the man page)
$ nc -lU /tmp/sock > filename.out      # Terminal #1 (sender)
$ nc -U /tmp/sock < filename.in        # Terminal #2 (receiver)





TCP Sockets (Ports)


TCP ports are very much like Unix sockets. Instead of a file, the data stream is buffered. And Unix sockets are not networkable, but TCP ports are a foundation of the Internet. Ports use client/server, and the server owns (binds) the port that the client connects to. Sockets and Ports use the same shell application (netcat) and Python3 module (socket).

Since there is no file to create or delete, bind() and connect() simply use an IP address and port number.

Processes using a TCP port usually work in the following order (just like a Unix Domain Socket):
  • Create a file descriptor (fd) that defines the port
  • Server binds to the port
  • Server listens on the port
  • Client connects to the port
  • Server accepts the connection
  • Both sides read/write
  • Client closes port
  • Server closes port (usually during service shutdown)
  • Delete/unlink the file descriptor and port (if it won't be reused)


TCP Port Services (Bind to port):

The server owns (binds) and monitors the port.

Shell:

$ nc -l 45678      # Port 45678

Python3:

>>> import socket
>>> tempsocket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
>>> tempsocket.bind(("127.0.0.1", 45678))   # Tuple: (IP Addr, Port)
>>> tempsocket.listen(1)
>>> conn, addr = tempsocket.accept()   # Listener blocks here awaiting sender's connect()
>>> conn.recv(1024).decode('UTF-8')    # Listener blocks here awaiting send()
'Hello, World'                                  # Try it a few times to catch each send()
>>> tempsocket.close()

C:

#include <stdio.h>
#include <string.h>
#include <netinet/in.h>
int main()
{
     int sock_fd, conn_fd;   // Socket file descriptor and Connection file descriptor
     int port_num = 45678;
     socklen_t clilen;
     char buffer[256];
     struct sockaddr_in serv_addr, cli_addr;
     sockfd = socket(AF_INET, SOCK_STREAM, 0);        // Define socket type
     memset(&serv_addr, 0, sizeof(serv_addr));    // Set array to zeroes 
     serv_addr.sin_family = AF_INET;
     serv_addr.sin_addr.s_addr = INADDR_ANY;
     serv_addr.sin_port = htons(port_num);
     bind(sock_fd, (struct sockaddr *) &serv_addr, sizeof(serv_addr));   // Bind
     listen(sock_fd,5);                               // Listen
     clilen = sizeof(cli_addr);
     conn_fd = accept(sock_fd, 
                 (struct sockaddr *) &cli_addr, 
                 &clilen);                        // Accept
     memset(&buffer, 0, sizeof(buffer));          // Fill buffer with zeroes - prevent garbage
     read(conn_fd,buffer,255);                        // Read
     printf("Here is the message: %s\n",buffer);
     write(conn_fd,"I got your message",18);          // Write
     close(conn_fd);
     close(sock_fd);                                  // Close
     return 0; 
}


TCP Port Clients (Connect to port):

The client connects to the port on-demand.

Shell:

$ nc localhost 45678      # IP=loopback, Port #45678

Python3:

>>> import socket
>>> tempsocket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
>>> tempsocket.connect(("127.0.0.1", 45678))   # Tuple: (IP Addr, Port)
>>> data = "Hello, World".encode("utf-8")
>>> tempsocket.send(data)          # Try this a few times
12
>>> tempsocket.close()

C:

#include <stdio.h>
#include <string.h>
#include <netdb.h>
int main(int argc, char *argv[])
{
    int sock_fd;
    int port_num = 45678;
    struct sockaddr_in serv_addr;
    struct hostent *server;
    char buffer[256];
    sock_fd = socket(AF_INET, SOCK_STREAM, 0);       // Define socket type
    server = gethostbyname(port_num);                // Lookup port owner
    memset(&serv_addr, 0, sizeof(serv_addr));    // Set array to zeroes
    serv_addr.sin_family = AF_INET;
    bcopy((char *)server->h_addr,
         (char *)&serv_addr.sin_addr.s_addr,
         server->h_length);
    serv_addr.sin_port = htons(port_num);
    connect(sock_fd,(struct sockaddr *) &serv_addr,sizeof(serv_addr)); // Connect
    printf("Please enter the message: ");
    memset(&buffer, 0, sizeof(buffer));
    fgets(buffer,255,stdin);                // Keyboard input
    write(sock_fd,buffer,strlen(buffer));   // Write to server
    memset(&buffer, 0, sizeof(buffer)); // Reuse buffer array - clear to prevent garbage
    read(sock_fd,buffer,255);               // Read response
    printf("%s\n",buffer);                  // Print response
    close(sock_fd);                         // Close
    return 0;
}


More fun with netcat

This is the basic way that webservers work.

$ echo "Hello, World" | nc -l 45678   # Enter in Terminal #1
$ nc localhost 45678                  # Enter in Terminal #2
                                      # OR open http://localhost:45678
                                      # in your favorite web browser

Monday, June 3, 2013

Pre-filling an Outlook Express mail in Windows XP

I want a Python3 script to open a pre-filled Outlook Express (OE) new-mail window. A human should read, edit, and approve the e-mail before sending. If I wanted a completely automatic e-mail, I would use Python's smtp module.

The problem is that OE has no COM server, and the Simple-MAPI support is in C++.




Command Line

There is an easy way from the Windows command line:

C:>"C:\Program Files\Outlook Express\msimn" /mailurl:mailto:"name%40example.com?cc=cc1%40example.com&bcc=bcc1@example.com&subject=The%20Subject%20Line&body=The%20first%20paragraph.%0A%0AThe%20second%20paragraph.%0A%0AThanks%2C%0AName

This launches OE with a single argument. That argument is a mailto: URL. The URL includes to:, cc:, bcc:, subject:, and the body.
  • "C:\Program Files\Outlook Express\msimn" launches OE
  • /mailurl:mailto:" defines the mailto: URL. The quote(") is required on the command line, though not in the python version.
  • name%40example.com? is the To: line. The To: line ends with the question-mark (?)
  • cc=cc1%40example.com& the rest of the fields are defined (cc=) and separated with the ampersand (&).
  • Don't use spaces or commas or returns - just text.
    space%20
    ,%2C
    return%0A
    @%40
  • I have not figured out a way to include more than one e-mail address on each line.
  • Files cannot be attached using this method.




Python

Since OE does not have any COM or other programmatic way to do it, we will fall back on subprocess, and have Python simply create a mailto: URL and use the command line.

This simple Python3 script creates a pre-filled Outlook Express window using subprocess:


import subprocess

# The header and body
to      = "name1%40example.com"
cc      = "cc1%40example.com"     # or leave blank: cc = ""
bcc     = "bcc1%40example.com"    # or leave blank: bcc = ""
subject = "The%20Subject"
body    = "The%20first%20paragraph.%0A%0AThe%20second%20paragraph.%0A%0AThanks%2C%0AName"

oe = "C:\Program Files\Outlook Express\msimn"
mailto_url = ('/mailurl:mailto:' + to      + '?' +   # No lone quote (") after colon (:)
              'cc='              + cc      + '&' +
              'bcc='             + bcc     + '&' +
              'subject='         + subject + '&' +
              'body='            + body )            # No ending ampersand (&) 

subprocess.call([oe, mailto_url])

Tuesday, May 7, 2013

Brainstorm Big 5 - May 2013

These are a few of the more interesting current ideas on Brainstorm. When you prepare for UDS, also do a quick Brainstorm search. It's a good way to see what users have thought about the topic in the past.

If you want to leave Developer Feedback on a Brainstorm Idea, but lack permissions, then please contact me. I can assign you permission, or simply add your response for you.  (ian dash weisser at ubuntu dot com)




The all-time top four:

The current top four all-time Brainstorm Ideas. They change from time to time as new ideas overtake old ideas, or as ideas get merged or implemented or closed.

You may see a common theme among them:

1) Restoring the bootloader by Ubuntu Live media
2) Graphical frontend to edit GRUB menu
3) Provide a simple interface for labeling partitions and external drives
4) Better Hardware Profile Manager

Here is what I see: None of these seem like features requested by unskilled users.

Instead, these seem more likely to be used by migrating power-users who have imprinted upon previous systems...and then discovered their first hurdle on the learning curve.

Now, I'm not proposing that we should implement any of these ideas. Instead, consider it a data point - here is one measure of how Ubuntu is perceived by rather skilled new users. Not what they actually need to be productive, but what they spend their time looking for fruitlessly.

And when they don't find it, some of them rant about Ubuntu. Goodness, just look at some of those comments.
  • Do we want these issues to be the first hurdles for this type of user?
  • Is there an easy alternative we can draw them into?
  • Is there a better message that Ubuntu, Launchpad, the forums, the Teams, etc. should be communicating to them?
  • Are these opportunities to begin their learning curve in a kinder, gentler way?
  • I wonder why those users, after overcoming the hurdle, did not implement a solution to help those who came after them?






Monetization by committee

To round out the Big 5 this month,

Alternative Sources of Income

The top Idea of the past six months, this is a crazy-quilt of monetization ideas.  I've been -among other things- a banker and a real-business-with-employees owner and a QA inspector, so I completely understand how unrealistic some of the money-handling-and-administration Solutions really are. But like the all-time top-four (above), the real message is the intent and the context.
  1. People are still really frustrated by [what they think are] bugs.
    Frustrated enough that some are willing to pay [small amounts] for bug bounties.
    Yet apparently not frustrated enough to actually get involved.

  2. There is also a lot of FUD still floating around about unity-lens-shopping, and some new users are still getting attracted to the tinfoil-hat crowd's message.
  • Is this an opportunity to recruit for the Bug Squad? LoCos? Teams? Upstreams?
  • How can we make our commitment to personal data privacy clearer?
  • How can we improve the message that Ubuntu users are participants, not customers?

Feel free to discuss in the appropriate Brainstorm Idea page, or in the comments section of this blog, or by e-mail...or anywhere else you like. Like at UDS.
Yeah, at UDS.
See you on IRC at UDS!

Saturday, May 4, 2013

Six ways to make a notification pop-up

UPDATED MAY 2013 to add Go and GObject methods to the existing Command Line and Python methods.
Originally published Feb 27, 2009.

There are several ways to create notifications in Ubuntu. The notify-OSD daemon draws the boxes and creates the notifications, so all these ways simply contact it and tell it what to do.

  1. On the command line, use the notify-send command. This command is part of the libnotify-bin package. It's easy:
    $ notify-send -i /usr/share/icons/Tango/32x32/status/sunny.png \
                       "Notification Title" \
                       "This is the body"

  2. More command line: dbus-send command does NOT work for notification. It does not support nested message variables. If it did, it should be something really close to this:
    dbus-send --session --dest=org.freedesktop.Notifications \
                             --type=method_call --reply-timeout=10000 \
                             /org/freedesktop/Notifications org.freedesktop.Notifications \
                             string:'Test Application' uint32:0 \
                             string: string:'Notification Title' \
                             string:'This is the body' \
                             array:string: dict:string: int32:10000

  3. Using the Python2 or Python3 pynotify module:
    #!/usr/bin/env python
    """Python 2.5 script. Creates a Notification pop-up bubble"""
    import pynotify
    title = "Notification Title"
    text  = "This is the body"
    icon  = "/usr/share/icons/Tango/32x32/status/sunny.png"
    pynotify.init("Test Application")
    notification = pynotify.Notification(title, text, icon) 
    notification.set_urgency(pynotify.URGENCY_NORMAL)
    notification.show() 

  4. Using the Python2 or Python3 dbus module:
    #!/usr/bin/env python
    """Python 2.5 script. Creates a Notification pop-up bubble"""
    import dbus
    item              = "org.freedesktop.Notifications"
    path              = "/org/freedesktop/Notifications"
    interface         = "org.freedesktop.Notifications"
    app_name          = "Test Application"
    id_num_to_replace = 0
    icon              = "/usr/share/icons/Tango/32x32/status/sunny.png"
    title             = "Notification Title"
    text              = "This is the body"
    actions_list      = ''
    hint              = ''
    time              = 5000   # Use seconds x 1000
    
    bus = dbus.SessionBus()
    notif = bus.get_object(item, path)
    notify = dbus.Interface(notif, interface)
    notify.Notify(app_name, id_num_to_replace, icon, title, text, actions_list, hint, time)

  5. Using the Python3 GObject Introspection (gi) module:
    #!/usr/bin/env python3
    import gi.repository
    from gi.repository import Gio, GLib
    
    session_bus                = Gio.BusType.SESSION
    cancellable                = None
    proxy_property             = 0
    interface_properties_array = None
    destination                = "org.freedesktop.Notifications"
    path                       = "/org/freedesktop/Notifications"
    interface                  = destination
    method                     = "Notify"
    application_name           = "Test Application"
    title                      = "Notification Title"
    text                       = "This is the body"
    id_num_to_replace          = 0
    icon                       = "/usr/share/icons/Tango/32x32/status/sunny.png"
    actions_list               = []
    hints_dict                 = {}
    display_milliseconds       = 5000
    timeout                    = -1
    
    connection = Gio.bus_get_sync(session_bus, cancellable)
    notify = Gio.DBusProxy.new_sync( connection, proxy_property, interface_properties_array,
                                     destination, path, interface, cancellable)
    args = GLib.Variant('(susssasa{sv}i)', ( application_name, id_num_to_replace, icon, 
                        title, text, actions_list, hints_dict, display_milliseconds))
    result = notify.call_sync(method, args, proxy_property, timeout, cancellable)
    id = result.unpack()[0]
    print(id)

  6. Using the go programming language (golang):
    package main
    
    import "launchpad.net/~jamesh/go-dbus/trunk"
    import "fmt"
    
    func main() {
        // Set the variables
        var (
            err              error
            conn             *dbus.Connection
            destination      string = "org.freedesktop.Notifications"
            path             string = "/org/freedesktop/Notifications"
            dbus_interface   string = "org.freedesktop.Notifications"
            dbus_method      string = "Notify"
            application_name string = "dbus-test"
            id_num_to_repl   uint32
            icon_path        string = "/usr/share/icons/Tango/32x32/status/sunny.png"
            title            string = "Notification Title"
            text             string = "This is the body"
            actions_list     []string
            hint             map[string]dbus.Variant
            timeout          int32  = -1
        )
    
        // Connect to Session or System buses.
        if conn, err = dbus.Connect(dbus.SessionBus); err != nil {
            fmt.Println("Connection error:", err)
        }
    
        // Create an object proxy
        obj := conn.Object(destination, dbus.ObjectPath(path))
    
        // Call object methods.
        reply, err := obj.Call(
            dbus_interface,             dbus_method,
            application_name,           id_num_to_repl,
            icon_path,                  notif_box_title,
            notif_box_body,             actions_list,
            hint,                       timeout)
        if err != nil {
            fmt.Println("Notification error:", err)
        }
    
        // Parse the reply message
        var notification_id uint32
        if err := reply.Args(&notification_id); err != nil {
            fmt.Println(err)
        }
        fmt.Println("Notification id:", notification_id)
    


Zenity is an application, not a daemon. It creates a regular window that can be abused for notifications.
zenity --info --title "Big Title" --text "This is the\\nbody text"

Sunday, March 3, 2013

Creating libraries and linking to libraries in Vala

This is an example of how to create a library in vala, and how to access it using several different methods. This example includes:
  1. A shared library written in vala, including
    • shared object (.so), and how to install it
    • headers (.h), and
    • bindings (.vapi)
  2. A command-line tool that uses the library by direct-linking
  3. A dbus server that uses the library by direct linking
    • makes the library available to other dbus-aware applications
    • self-terminating process after use (not a daemon)
  4. A command-line tool that uses the library via dbus and the dbus-server.

The first half of the example is based on the official vala tutorial library example,
I have added dbus connectivity and a bit more explanation that I found helpful.





1) Create an empty directory. These steps create a lot of files!

    $ mkdir test_library
    $ cd test_library



2) Create the library. Save this file as libtest.vala:

// BEGIN
public class MyLib : Object {

    public string hello() {
      return "Hello World from MyLib";
   }

   public int sum(int x, int y) {
      return x + y;
   }
}
// END



3) Create the .c, .h (C header), and .vapi (vala binding) files:
   -C --ccode   Output C code instead of compiled
   -H --header=file  Output C header file
      --library=name Assign name to library
      --vapi         Assign name to vapi file
   -b --basedir=dir  Use dir as the base source dir (For me, this is unnecessary)

    $ valac -C -H libtest.h --library libtest libtest.vala --basedir ./



4) Compile the library using:

   -shared       Create a .so shared object
   -fPIC         Position Independent Code (PIC) suitable for use in a shared library
   $(pkg-config --cflags gobject-2.0)
                 Output: -I/usr/include/glib-2.0
                         -I/usr/lib/i386-linux-gnu/glib-2.0/include
   -o filename   Output filename

     $ gcc -shared -fPIC -o libtest.so $(pkg-config --cflags --libs gobject-2.0) libtest.c



5) Create the command-line application that uses the library by linking.
   Save this file as hello.vala:

// BEGIN
void main() {
    var test = new MyLib();

    // MyLib hello()
    stdout.printf("%s\n", test.hello());

    // MyLib sum()
    int x = 4, y = 5;
    stdout.printf("The sum of %d and %d is %d\n", x, y, test.sum(x, y));
}
// END




6) Compile the command-line application.
Using vala, there need to be TWO links:
  • The vala link to a .vapi file (in this example, test.vapi)
  • The gcc link to a C library (in this example, -X -ltest to add libtest.so)

   - The vala link to a .vapi file (in this example, test.vapi)
   - The gcc link to a C library (in this example, -X -ltest to add libtest.so)

   -X --Xcc=-I.    Pass the -I. (include current directory) to gcc.
                   GCC will look for shared libraries in the current directory first
   -X --Xcc=-L.    Pass the -L. (add current directory to search path) to gcc.
   -X --Xcc=-ltest Link library "libtest" which we just created in the current directory. 

   If we had the libtest.so in the local directory, we could use:

    $ valac -X -I. -X -L. -X -ltest -o hello hello.vala libtest.vapi



7) Test the command-line application that uses the library:
The library is in the local directory (uninstalled):

     $ LD_LIBRARY_PATH=$PWD ./hello
     Hello World from MyLib
     The sum of 4 and 5 is 9



8) We cannot easily use the local library for dbus.
So install the library to the expected location.
Copy the library to /usr/lib using:

      $ sudo cp libtest.so /usr/lib/



9) Test the (installed) command-line application:

     $ ./hello
     Hello World from MyLib
     The sum of 4 and 5 is 9



10) Create the dbus server that uses the library.
A server listens for requests from other applications. This server acts as a gateway: It translates other application requests for library methods into a library call, and then sends the response back across dbus to the original requestor.

Save this file as dbus_server.vala:

// BEGIN
// Source: https://live.gnome.org/Vala/DBusServerSample#Server

[DBus (name = "org.example.Demo")]   // dbus interface name
public class DemoServer : Object {

    /* Functions that access the library on the Demo interface
     * Note the LACK of \n in the return strings.
     * Vala automatically mangles my_function_name into the
     *   standard Dbus camelcase MyFunctionName
     * So a function is "hello()" here, but "Hello" on Dbus 
     */
    public string sum () { 
        var test = new MyLib();
        int x = 4, y = 5;
        return "The sum of %d and %d is %d".printf( x, y, test.sum(x, y));
        // Note the LACK of \n in the return strings
    }

    public string hello () { 
        var test = new MyLib();
        return "%s".printf( test.hello());
    }

}

[DBus (name = "org.example.DemoError")]
public errordomain DemoError { SOME_ERROR }

/* Dbus functions */
void on_bus_aquired (DBusConnection conn) {
    try {
        string path = "/org/example/demo";
        conn.register_object (path, new DemoServer ());
    } catch (IOError e) { 
    stderr.printf ("Could not register service\n"); }
}

void main () {
    GLib.MainLoop loop           = new GLib.MainLoop ();
    GLib.TimeoutSource time      = new TimeoutSource(3000);   // 3 sec
    GLib.BusType bus             = BusType.SESSION;
    string destination           = "org.example.demo";
    GLib.BusNameOwnerFlags flags = BusNameOwnerFlags.NONE;

    Bus.own_name ( bus, destination, flags,
                  on_bus_aquired,
                  () => {},
                  () => stderr.printf ("Could not aquire name\n"));

    // Use timeout to quit the loop and exit the program
    time.set_callback( () => { loop.quit(); return false; });
    time.attach( loop.get_context() );  // Attach timeout to loop

    loop.run ();                // Listen for dbus connections

}
// END



11) Compile the dbus server:
   Dbus requires the gio package (--pkg gio-2.0)
   -X --Xcc=-I.    Pass the -I. (include current directory) to gcc.
                   GCC will look for shared libraries in the current directory first
   -X --Xcc=-L.    Pass the -L. (add current directory to search path) to gcc.
   -X --Xcc=-ltest Link library "libtest" which we just created in the current directory. 

   Since the library is installed, we can ignore those local directory flags:

      $ valac --pkg gio-2.0 -X -ltest dbus_server.vala libtest.vapi



12) Create a dbus .service file, so dbus knows how to find our new dbus_server.
Save this file as dbus.service. Your third line will vary - use the real path:

// BEGIN

[D-BUS Service]

Name=org.example.demo

Exec=/home/me/vala/library/dbus_server

// END



13) Install the dbus service file:

      $ sudo cp dbus.service /usr/share/dbus-1/services/org.example.demo.service



14) Test the dbus server using an existing command-line application, dbus-send.
This is a command-line application using the library via dbus, using dbus_server for it's intended purpose as a gatetay.

The sender is different each time because dbus-send creates a new process and connection each time it is used.
The destination is different because the server terminates after 3 seconds.

      $ dbus-send --session --type=method_call --print-reply \
        --dest="org.example.demo" /org/example/demo org.example.Demo.Hello
      method return sender=:1.3173 -> dest=:1.3172 reply_serial=2
      string "Hello World from MyLib"

      $ dbus-send --session --type=method_call --print-reply \
        --dest="org.example.demo" /org/example/demo org.example.Demo.Sum
      method return sender=:1.3175 -> dest=:1.3174 reply_serial=2
      string "The sum of 4 and 5 is 9"


15) Clean up:
Remove the dbus service file:
Remove the test library from /usr/lib

     $ sudo rm /usr/share/dbus-1/services/org.example.demo.service
     $ sudo rm /usr/lib/libtest.so

Thursday, February 21, 2013

Logging with Vala

How to send messages to syslog using Vala:

// log.vala

public static void main() {

    // Application Name (called 'domain' in Gspeak)
    string application_name = "MyTestApplication";

    // Log message
    string message = "This is a test message";

    // Log level - set to "Warning"
    GLib.LogLevelFlags glib_level       = GLib.LogLevelFlags.LEVEL_WARNING;
    int                posix_level      = Posix.LOG_WARNING; 

    // Print to stderr, not syslog or another log
    GLib.log( application_name, glib_level, message );

    // Log to syslog using the posix bindings
    // Posix.LOG_PID and Posix.LOG_USER are defined by Posix, not me.
    Posix.openlog(application_name, Posix.LOG_PID, Posix.LOG_USER);
    Posix.syslog(posix_level, message);

    return;
}

Compile using valac --pkg posix log.vala

String Arrays in Vala

String Arrays are simply bunches of strings grouped together

string[] apples = {"red delicious", "granny smith", "macintosh", "gala", "fuji"}

They look like Python lists, but they are not. They often don't work like lists (you cannot slice them), but they work *really* fast. They are useful if you have a data set that is (mostly) immutable...you can append to it, but I haven't found a way to delete strings from the array without copying everything else into a new array.

Here is a demo program showing how to
  • Create a string array
  • Append a string to an array
  • Match a string in an array
  • Determine the index of the matched string
  • Retrieve a string from the array (non-destructively)
  • Replace a string with another withing an array
  • Concatenate the array into a single string (for printing)

// string_array.vala

// If a in b
void if_a_in_b ( string a, string[] b, string b_name )  {
    if (a in b) { 
        stdout.printf("Found %s in %s\n", a, b_name); 
    }
    else {
        stdout.printf("%s not Found in %s\n", a, b_name);
    }
    return;
}


// One way to print an array
void print1 (string[] a, string a_name) {
    stdout.printf("Array %s: ", a_name);
    foreach (string item in a) {
        stdout.printf("%s, ", item);
    }
    stdout.printf("\n");
    return;
}

// Another way to print an array
void print2 (string[] a, string a_name) {
    string a_string = string.joinv(", ", a);
    stdout.printf("Array %s : %s\n", a_name, a_string);
    return;
}

// Index of an item in an array
void index_array (string[] a, string a_name, string match) {
    // Record the index of all matches
    int[] indexes = {};
    int i;
    for (i = 0; i < a.length; i++) {
        if (a[i] == match) {
            indexes += i;
        }
    }
    // Print the results
    if (indexes.length == 0) {
        stdout.printf("Indexing %s: %s not found\n", a_name, match);
    }
    else if (indexes.length == 1) {
        stdout.printf("Indexing %s: %s found at position %d\n", 
        a_name, match, indexes[0]);        
    }
    else if (indexes.length == 2) {
        stdout.printf("Indexing %s: %s found at positions %d and %d\n", 
        a_name, match, indexes[0], indexes[1]);        
    }
    else {
        stdout.printf("Indexing %s: %s found at positions ", 
        a_name, match);
        // Convert ints to a strings
        int j;
        for (j = 0; j < indexes.length; j++) {
            if ( j < (indexes.length - 1)) {
                stdout.printf("%d, ", indexes[j]);
            }
            else {
                stdout.printf("and %d.\n", indexes[j]);
            }
        }
    }
    return;
}



void arrays () {
    // Create two string arrays 
    string[] orange = { "fred", "joe", "allen", "steve" };
    string[] blue   = { "jane", "sam", "ellie", "terri" };

    // Test contents of each array
    if_a_in_b ("fred", orange, "Orange");
    if_a_in_b ("fred", blue, "Blue");

    // Length of an array
    stdout.printf("List length: %i\n", orange.length);

    // Item from an array
    stdout.printf("Orange item #2: %s\n", orange[1]);

    // Replace one string in an array
    blue[2] = "pamela";

    // Determine the index (location) of a string in an array
    index_array(blue, "blue", "pamela");

    // Append one string to an array 
    blue += "stacy";

    // Print an array
    print1(orange, "orange");
    print2(blue, "blue");

    return;
}


// Main
public static void main() {
    arrays ();
    stdout.printf("Hello, World\n");
    return;
    }


Compile with a simple valac string_array.vala



Thursday, February 14, 2013

Using custom C libraries with Vala

Let's try to use Vala with custom C libraries.

For this example, let's see if we can use Vala to manilulate a gdbm database.
Gdbm is a tiny db used in lots of applications.

In order for a C library to work in Vala, it needs a .vapi file to translate.
In this example, we will use the gdbm library (a small, common database in C)  using Vala bindings.

There are two issues for new users here:
  1. Linking to C libraries in the C compiler
  2. Linking to vapi files in the Vala interpreter
You'll see how to handle both.


1) Download the sources:
  • Create a working directory
  • Install the libgdbm-dev package, to get the c headers.
  • I discovered a vapi file for gdbm created by Andre Masella. Thank you, Andre!
  • Download and install the vapi file. Vapi files belong in /usr/share/vala/vapi/
  • Open a browser window to the Gnu GDBM documentation.
$ mkdir /home/me/vala/gdbm_vala
$ cd /home/me/vala/gdbm_vala
# sudo apt-get install libgdbm-dev
$ wget https://raw.github.com/apmasell/vapis/master/gdbm.vapi
# sudo ln /home/me/vala/gdbm_vala/gdbm.vapi /usr/share/vala/vapi/


2) Create a simple file in C to test the gdbm library:
This simple file (source) is /home/me/vala/gdbm_vala/gdbm_version1.c
It gets the gdbm_version string from the gdbm library.
#include <stdio.h>
#include <gdbm.h>
int main() {
    printf("VERSION (C): %s.\n", gdbm_version);
}
Let's compile it and run it to test the c headers we retrieved in the libgdbm-dev package.
  • The gcc -o flag specifies the output file name

$ gcc -o gdbm_version1 gdbm_version1.c
/tmp/cc31QKu0.o: In function `main':
gdbm_version1.c:(.text+0xa): undefined reference to `gdbm_version'
collect2: error: ld returned 1 exit status

Uh-oh. Fatal error.
The compiler did not understand the 'gdbm_version' variable because gdbm.h is not in it's standard library. I must tell the compiler to link to it using the -l flag.

Try again, linking to the gdbm library.

$ gcc -o gdbm_version1 gdbm_version1.c -l gdbm
$ ./gdbm_version1
VERSION (C): GDBM version 1.8.3. 10/15/2002 (built Jun 11 2012 00:27:26).

There should be no errors or warnings. Success!


3) First program:

Here's the relevant entry in the vapi file for the version number:

[CCode(cheader_filename = "gdbm.h")]
namespace GDBM {
    ...
    [CCode(cname = "gdbm_version")]
    public const string VERSION;
    ...
}

Namespace GDBM + string VERSION, so "string GDBM.VERSION" in Vala should translate to "string gdbm_version" in C. And we just tested that the C version works.

Let's try a simple Vala program that checks the version number:

private static void main () {
    stdout.printf("The gdbm version string: %s\n", GDBM.VERSION);
    }

And compile it:
  • --pkg gdbm tells valac to look for the gdbm vapi file.

$ valac --verbose --pkg gdbm gdbm_version2.vala 
Loaded package `/usr/share/vala-0.18/vapi/glib-2.0.vapi'
Loaded package `/usr/share/vala-0.18/vapi/gobject-2.0.vapi'
Loaded package `/usr/share/vala/vapi/gdbm.vapi'
cc -o '/home/ian/vala/gdbm_vala/code/gdbm_version2' '/home/ian/vala/gdbm_vala/code/gdbm_version2.vala.c' -I/usr/include/glib-2.0 -I/usr/lib/i386-linux-gnu/glib-2.0/include  -lgobject-2.0 -lglib-2.0
/tmp/ccnYcsLH.o: In function `_vala_main':
gdbm_version2.vala.c:(.text+0xf): undefined reference to `gdbm_version'
collect2: error: ld returned 1 exit status
error: cc exited with status 256
Compilation failed: 1 error(s), 0 warning(s)

Uh-oh. Something went wrong.

When something goes wrong , the --verbose flag is your friend.
The output shows that the Vala was translated into C without any warnings or errors, but then something was wrong when the compiler tried to prcoess the C.

Wait a second...it's the SAME ERROR we had before in the C program!
We know how to fix that: Tell the compiler to link (-l flag) to the gdbm library.

Check the compiler command, and --sure enough-- link to the gdbm library is missing.

Two lessons here:
  • Vala's link to a pkg does not necessarily mean the compiler is linked to the library. valac --pkg foo may not translate into cc -lfoo
  • Use valac's -Xcc flag to pass links to the compiler.

Let's try compiling again, adding /usr/include/gdbm.h using an Xcc flag, and removing --verbose:
  • --pkg gdbm tells valac to use the gdbm vapi file
  • --Xcc=-gdbm tells the C compiler to link to the C gdbm header

$ valac --pkg gdbm --Xcc=-lgdbm gdbm_version2.vala 
$ ./gdbm_version2
The gdbm version is: GDBM version 1.8.3. 10/15/2002 (built Jun 11 2012 00:27:26)

No errors or warnings. Success!


4) Take a break

We are now past the a bunch of biggest beginner hurdles:
  1. How to figure out common compile errors.
  2. How to tell the compiler to link to libraries.
  3. How to tell valac to use vapi files.

 5) More complicated program

Here is a more complicated program that opens a test database (or create a new db if it doesn't exist), and look for a certain key. If the key is not in the database, it appends the key:value pair to the database.
  • It uses the gdbm.vapi bindings for open(), read/write/create permission, contains(), and save() [which really means append].

// File: /home/me/vala/gdbm_vala/code/gdbm_add
private static int main () {

    // Hard-coded values
    string filename = "/home/me/vala/gdbm_vala/code/sample_db.gdbm";
    string key_string = "spam";
    string value_string = "eggs";

    // Open the database
    // GDBM.OpenFlag.WRCREAT comes from the vapi file. It means read+write+create new db
    GDBM.Database db = GDBM.Database.open (filename, 0, GDBM.OpenFlag.WRCREAT);

    // Convert the string values to bytes. gdbm doesn't save strings.
    uint8[] key_int = key_string.data;
    uint8[] value_int = value_string.data;

    // If the key_int is already in the database, say so and exit.
    if (db.contains (key_int) == true) {
        stdout.printf("In db\n");
        return 0;
        }
    stdout.printf("Not in db\n");

    // If appending the key:value pair to the database is successful, say so and exit.
    if (db.save (key_int, value_int, false) == true) {
        stdout.printf("Good append saved\n");
        return 0;
        }

    // Problems
    stdout.printf("Failed to append\n");
    return 1;
    }

Let's compile it.

$ $ valac --pkg gdbm --Xcc=-lgdbm gdbm_add.vala 
/home/ian/vala/gdbm_vala/code/gdbm_add.vala.c: In function ‘_vala_main’:
/home/ian/vala/gdbm_vala/code/gdbm_add.vala.c:192:2: warning: passing argument 1 of ‘gdbm_open’ discards ‘const’ qualifier from pointer target type [enabled by default]
In file included from /home/ian/vala/gdbm_vala/code/gdbm_add.vala.c:9:0:
/usr/include/gdbm.h:85:18: note: expected ‘char *’ but argument is of type ‘const gchar *’

Oh, my.

Let's look more closely at these error messages. Each error takes two lines
  • The first warning is in the generated C file, line 192, warning: passing argument 1 of ‘gdbm_open’ discards ‘const’ qualifier from pointer target type [enabled by default].

    Line 192 of the C file looks like _tmp4_ = gdbm_open (_tmp3_, 0, GDBM_WRCREAT, 0644, NULL); which looks a lot like the original gdbm_open function again.

    Valac is moving the filename to _tmp3_, and that _tmp_ variable seems to be the wrong type. It's just a warning - the code still compiles. And it seems to be a bug in valac, not a problem with our code.
  • The second warning is in the generated C file, line 9. That's the #include line. The warning is expected ‘char *’ but argument is of type ‘const gchar *’

    This is an interesting warning, and I'm not sure my merely including the header would trigger it...but it's also just a warning, and the compiled binary works.
Finally, let's run the binary a few times. The first time, the database does not exist, so the program should create the database and populate it with one key:value pair. On subsequent runs, the database should already exist, and the program should successfully find the existing key.

$ ./gdbm_add 
Not in db
Good append saved
$ ./gdbm_add 
In db
$ ./gdbm_add 
In db

It works!

We've gotten past the namespace hurdle, the vapi hurdle, and have successfully manipulated a gdbm database using the original C library.

Saturday, February 9, 2013

Starting to learn Vala

I have decided to learn Vala. Instructions are here.




Hello Dbus:

Everyone starts with "Hello World." I did too. Little is gained by repeating it. So here is something different. Here is my first dbus introspection in vala:

// This vala file is named dbus-introspection.vala
[DBus (name = "org.freedesktop.DBus.Introspectable")]   // Dbus interface
interface Introspectable : Object {
    [DBus (name = "Introspect")]                        // Dbus method
    public abstract string introspect() throws IOError;
}

void main () {
    Introspectable conn = null;    // If inside the try, causes fail
    try {
        conn = Bus.get_proxy_sync (BusType.SESSION,    // Dbus session
                   "org.freedesktop.Notifications",    // Dbus destination
                   "/org/freedesktop/Notifications");  // Dbus path
        string reply = conn.introspect();              // the actual event
        stdout.printf ("%s\n", reply);                 // print response
    } catch (IOError response) {                       // print error
        stderr.printf ("IO Error: %s\n", response.message);
    }
}

Compile using $ valac --pkg gio-2.0 dbus-introspect.vala
Run using $ ./ dbus-introspect

The vala file is 19 lines.
The vala-generated C file is 300 lines, and about 15 times the size of the vala file.
The compiled binary is 19.4K, 130% the size of the C file.
But 19.4K is still pretty small, and it does run really fast!

For my own reference:

  • The dbus server information we need to connect. This is a common service that has Introspection:
    bus         = session bus
    destination = org.freedesktop.Notifications
    path        = /org/freedesktop/Notifications
    interface   = org.freedesktop.DBus.Introspectable
    method      = Introspect
    message     = (none)

  • main () { } is where the program starts.
    Equivalent to python's if "__name__" == "__main__":.
    Since it's the main, it returns either a return code (int) or nothing (void), and nothing else. The example above returns nothing.
    To add a return code, add return 0; to the end of the try { } block, and add return 1; to the end of the catch { } block

  • Michael Brown pointed out a very useful tidbit to avoid a lot of confusion:

  • Vala class = DBus interface
  • Vala interface = DBus proxy
  • This is the tricky part.
    Building the dbus connection (proxy) means we call that variable an interface in vala.
    And each dbus interface must be defined as a separate class in vala.
    To that, I will add that each class must be a GLib.Object.

    So a connection must start by opening an instance of the class, then adding the proxy information. Interface first, then proxy. That's backwards from the way dbus-send or python do it.

    For example:
    The dbus interface org.freedesktop.DBus.Introspectable gets defined as a vala class:
    interface Introspectable : Object { }


    Next, let's add the method and the labels to this class:
    [DBus (name = "org.freedesktop.DBus.Introspectable")]   // Dbus interface
    interface Introspectable : Object {
        [DBus (name = "Introspect")]                        // Dbus method
        public abstract string introspect() throws IOError;
    }
    

  • Methods must be public and abstract. We expect it to return a string (introspection returns a string of XML). As far as I can tell, methods should throw IOError in case the dbus message is malformed, the receiving service does not exist or has no such message, etc. The method is part of the interface's Object.


  • Main uses the try { } catch { } blocks to locate those IOErrors.
    Initial creation of the interface must be outside the try block, or the program won't be listening for the dbus response. Adding the bus and proxy information within the try block is not required...if you never make spelling mistakes.




Thursday, January 17, 2013

Logging in Python 3

Here is one way to set up proper logging in Python.

This sample script sends some messages to syslog and others to a different logfile based upon their priority.


#!/usr/bin/python3

# Import the Standard Library modules we need
import logging
import logging.handlers

def set_up_logging():

    # File handler for /var/log/some.log
    serverlog = logging.FileHandler('/var/log/some.log')
    serverlog.setLevel(logging.DEBUG)
    serverlog.setFormatter(logging.Formatter(
        '%(asctime)s %(pathname)s [%(process)d]: %(levelname)s %(message)s'))

    # Syslog handler
    syslog = logging.handlers.SysLogHandler(address='/dev/log')
    syslog.setLevel(logging.WARNING)
    syslog.setFormatter(logging.Formatter(
        '%(pathname)s [%(process)d]: %(levelname)s %(message)s'))

    # Combined logger used elsewhere in the script
    logger = logging.getLogger('wbs-server-log')
    logger.setLevel(logging.DEBUG)
    logger.addHandler(serverlog)
    logger.addHandler(syslog)

    return logger

logger = set_up_logging()
logger.debug('This message should go to the log file')
logger.info('So should this')
logger.warning('And this, too')


Friday, January 4, 2013

Replacing a bad drive in a RAID 1 array

About a month ago, I started getting daily mail from mdadm (the service that creates RAID arrays in software) that there was a degraded-array problem:

Subject: DegradedArray event on /dev/md0:myserver

This is an automatically generated mail message from mdadm
running on myserver

A DegradedArray event had been detected on md device /dev/md0.


Uh-oh. This is one reason why I don't use RAID for booting the system. I use a totally separate hard drive for the OS. My RAID is for data.
Let's login as root and take a look:

# mdadm --detail /dev/md0

/dev/md0:
        Version : 1.2
  Creation Time : Thu Mar 17 10:42:16 2011
     Raid Level : raid1
     Array Size : 1171873823 (1117.59 GiB 1200.00 GB)
  Used Dev Size : 1171873823 (1117.59 GiB 1200.00 GB)
   Raid Devices : 1
  Total Devices : 3
    Persistence : Superblock is persistent

    Update Time : Fri Jan  4 07:25:43 2013
          State : active, degraded
 Active Devices : 1
Working Devices : 1
 Failed Devices : 0
  Spare Devices : 0

           Name : myserver:0
           UUID : b8d85003:03eb3cd0:fbb98516:361fb411
         Events : 6596

    Number   Major   Minor   RaidDevice State
       0       8        1        0      active sync   /dev/sdb1
       1       0        0        1      faulty removed

So one of my RAID drives has apparently desynced, gone bad, or otherwise fallen off the array beyond the ability of mdadm to fix. sdb1 is the remaining good drive.

# blkid | grep linux_raid_member
/dev/sda1: UUID="b8d85003-03eb-3cd0-fbb9-8516361fb411" LABEL="myserver:0" TYPE="linux_raid_member" 
/dev/sdb1: UUID="b8d85003-03eb-3cd0-fbb9-8516361fb411" LABEL="myserver:0" TYPE="linux_raid_member" 

My two RAID drives are /dev/sda1 and /dev/sdb1. sdb1 is good. Therefore the faulty drive is sda1. Let's get the serial number of the bad drive, since that is printed on the outside of the drive.

# hdparm -I /dev/sda1 | grep "Serial Number"
 Serial Number:      WD-WMAZA3691591

Alternately, I could look for the serial number of the good drive(s).
Let's try re-adding the bad drive. Perhaps it was a onetime anomaly

# mdadm --add /dev/md0 /dev/sda1

Watch the progress if the sync using cat /proc/mdstat. For a 1200GB partition, my system takes about three hours for the mirroring to complete.

Often, re-adding the drive to the array is all you need to do, and the nonfaulty drive will happily rejoin it's friends in RAID.



However in this case, an hour later, the mirroring failed. Another degraded-array email showed up in my inbox. And a basic information command like hdparm -I /dev/sda1 failed with an input/output error. The drive is no longer trustworthy and must be replaced.

I ordered two slightly larger drives, to increase size to a three disk array. Replacement drives must be the same size or larger. I happen to know I have space in the server case for an additional drive, and that I have additional power connections and motherboard data connections available (and cables) for the new drive.

With the new drives in hand, it's time to poweroff the server (er, don't forget to tell network users that you're doing this!). Since we're adding and removing drives, there is a good chance that we will need access to BIOS...so that means moving the server to the desk and hooking up a keyboard and monitor.

First, remove the bad drive. Good thing we know the serial number!
Second, add the new drives. Test them by booting into BIOS and ensuring they are detected. While in BIOS, make sure the boot drive is correct. Mine had changed and I had to fix it.

Finally, boot the machine and gain root.

# ls /dev | grep sd
sda
sda1
sdb
sdb1
sdb2
sdb5
sdc
sdd

It looks like sda1 is our raid partition, sdb* is our non-raid system disk, and sdc and sdd are the new unpartitioned drives. We can verify that using parted

# parted -l
Model: ATA WDC WD20EARS-00M (scsi)
Disk /dev/sda: 2000GB
Sector size (logical/physical): 512B/512B
Partition Table: gpt

Number  Start   End     Size    File system  Name       Flags
 1      17.4kB  1200GB  1200GB  ext3         Server

Model: ATA WDC WD3200AVJB-6 (scsi)
Disk /dev/sdb: 320GB
Sector size (logical/physical): 512B/512B
Partition Table: msdos

Number  Start   End    Size    Type      File system     Flags
 1      1049kB  319GB  319GB   primary   ext3            boot
 2      319GB   320GB  1546MB  extended
 5      319GB   320GB  1546MB  logical   linux-swap(v1)

In order to add the new drives to the RAID array, we must first create a partition table (I chose GPT becasue these are large drives). Next, we partition them to match the existing drive. Then we can add them to the array.

# parted /dev/sdc mklabel gpt
# parted -a optimal /dev/sdc mkpart 1 ext3 17.4kb 1200GB
# mdadm --add /dev/md0 /dev/sdc1

# parted /dev/sdd mklabel gpt
# parted -a optimal /dev/sdd mkpart 1 ext3 17.4kb 1200GB
# mdadm --add /dev/md0 /dev/sdd1

See how each partition has three commands? See how we took the partition sizes directly from the existing partition above?

And now we are back in familiar territory - we have added drives to an array before. We can monitor the process using cat /proc/mdstat, and expect it to take a few hours.

And the array is humming along smoothly again.