London Open Source Jam 13

Thursday, July 2, 2009 at 4:10 PM

Here in London, we recently hosted London Open Source Jam 13 (Our 14th jam, as naturally we count from zero.) - unlucky for some, but not for us! We threw the floor open to talks of any kind and we had a bumper crop of lightning talks on a diverse range of topics. We learned how to make a wiki in 58 lines of python, why open source developers should care about open standards, some new approaches to database design, and a whole lot more.

Kai Hendry talked about Webconverger, a teeny weeny Linux Live CD designed for web kiosks, and his experience commercialising an open source project by offering services to go with it.

Zak Cohen shared his experiences using open source libraries in the games world in developing the award-winning game Climbactic. It turns out there's some great open source stuff out there, but sometimes paying for support is the only way to get the features you need.

Frederik Dohr and Mike Mahemoff spoke about TiddlyWeb, a generic RESTful store for structured data, and Scrumptious, a jQuery-based web app that allows people to annotate and comment on web pages, which uses TiddlyWeb for storage.

Other contributors included:
The OSJam website has more information about all the talks, and the photographs taken at the event are available on Picasa Web Albums.

We'll be planning the next OSJam for a couple of months' time - subscribe to the London Open Source Jam Group or keep any eye on the OSJam website (Atom feed) for more details.

By Malcolm Rowe and Matt Godbolt, Software Engineering Team

Google Update, regularly scheduled

Wednesday, July 1, 2009 at 3:00 PM

Client software products have unique challenges, with one of the important challenges being keeping the product up to date. Deploying critical security fixes in a timely and effective manner is one key way that we help keep users secure. Another benefit is that cool new features get deployed and adopted quickly. Google Update is the shared updating infrastructure used by Google Chrome, Google Earth and other Google software that runs on PCs to keep our products up to date on users' computers.

A couple of months ago we released the Google Update source code to give users and developers transparency into our update mechanism. One month later, we released Update Controls that allowed network administrators and advanced users the ability to control the installation and updating of Google products via Google Update.

We hope to help address concerns users have voiced around Google Update running as a continuous process. Until now, Google Update would always run in the background, functioning primarily as a reliable scheduler performing update checks at periodic time intervals. With today's release, Google Update now uses the Windows Task Scheduler to only run at periodic intervals. We have worked hard to ensure that automatic updates work just as reliably, and that our users are just as safe and secure as before.

There are a couple of details that we want to mention. First, in a very small number of cases when Google Update determines that the Windows Task Scheduler or Service mechanisms are not working as expected, we have added in fallback mechanisms that cause Google Update to begin running as a continuous process again to ensure users are still receiving updates to their Google software. Second, if you opted in to sending anonymous usage statistics and crash reports to Google for a particular Google application, we will run a process in the background called GoogleCrashHandler.exe. GoogleCrashHandler.exe is responsible for reporting crashes to Google when they occur in your Google product.

by S. Ganesh, Google Update Team

Introducing Apache Commons Math SimplexSolver

at 10:00 AM

SimplexSolver is an easy-to-use, object-oriented method of solving linear programming problems. We're happy to announce today that we've Open Sourced the code that runs the newly released Google Spreadsheets Solve Feature and made it a part of Apache Commons Math 2.0.

While numerous other libraries are available that run linear optimization problems, SimplexSolver is the first written in Java with a commercially-friendly license.

Let's say we have the following LP:

MIN -2x + y - 5
S.T.
x + 2y <= 6
3x + 2y <= 12
y >= 0

We could solve the problem in Java using the SimplexSolver:

// describe the optimization problem
LinearObjectiveFunction f = new LinearObjectiveFunction(new double[] { -2, 1 }, -5);
Collection constraints = new ArrayList();
constraints.add(new LinearConstraint(new double[] { 1, 2 }, Relationship.LEQ, 6));
constraints.add(new LinearConstraint(new double[] { 3, 2 }, Relationship.LEQ, 12));
constraints.add(new LinearConstraint(new double[] { 0, 1 }, Relationship.GEQ, 0));

// create and run the solver
RealPointValuePair solution = new SimplexSolver().optimize(f, constraints, GoalType.MINIMIZE, false);

// get the solution
double x = solution.getPoint()[0];
double y = solution.getPoint()[1];
double min = solution.getValue();

Looking at the LP problem and Java code side-by-side, we can see how easy it is to describe the problem in the Apache Commons Math API. More examples can be viewed in the SimplexSolverTest source code. Apache Commons Math 2.0 is not quite released yet, so for the time being if you want to use the SimplexSolver, you'll need to compile it from source.

So that's SimplexSolver: a clean, fast method of solving LP problems. We hope you like it as much as we do! Thanks to Luc Maisonobe and the rest of the Apache Team who helped integrate the SimplexSolver into the Apache codebase.

Australia Goes Open

Tuesday, June 30, 2009 at 2:30 PM



Here in Google's shiny new Sydney office, we recently hosted the first hackfest for OpenAustralia.org. OpenAustralia takes the data from Australia's record of Federal Parliament speeches, the Australian Hansard, and makes it easy for people to follow topics they're interested in.

Back in 2004, Matthew Landauer and Katherine Szuminska found themselves at the launch of the UK site TheyWorkForYou.com, a website designed to allow the average person to quickly and easily search the United Kingdom's Hansard - the record of all happenings in Parliament. Their work blossomed and they were inspired to make a similar site happen in Australia, OpenAustralia.org.

Both websites give ordinary people, who often have no idea who their local representative is, let alone what their representative has been doing on their behalf, the ability to track topics they're interested in and find out exactly what their representatives are doing.

When Matthew emailed out asking for a location for a place to host a hackfest, we were very happy to lend a helping hand. The hackfest ran on Saturday the 13th of June at the newly opened Google offices in Sydney, Australia and was successful beyond anyone's expectations.

When originally announced, the event's original 25 attendee slots filled so fast that we ended up increasing the number to 50 (which also filled remarkably fast). More surprisingly, almost everyone turned up and we even had a number of attendees fly in from other Australian states. No one expected this level of enthusiasm from the community, and we were pleased to share in everyone's excitement.



Some cool outcomes include a tool for crowd sourcing transcription of the "register of member's interests", an API and datasource for mapping street addresses to representatives, a "FixMyStreet" iPhone App and numerous bugs on the site fixed. More information on these and the ongoing efforts of this community can be found on the Open Australia Development list.



Matthew was also gracious enough to give a tech talk for Googlers about some of the many challenges faced by creating such a site, e.g. dealing with Crown Copyright, problems with getting a clean source of data, and problems getting the data fixed when it's clearly in error. Enjoy!



Guten LinuxTag!

Wednesday, June 24, 2009 at 6:03 PM

The Open Source Team's Cat Allman will be in Berlin, Germany this week to present at LinuxTag 2009. Cat's talk, Getting Started in Open Source: An Overview for Newbies, will take place on Friday, June 26th at 4 PM local time. This will be Cat's second time presenting at LinuxTag - in 2008 she gave a talk about Google Summer of Code™.

This year LinuxTag expects over 10,000 visitors from around the world- If you are there make sure to attend Cat's talk!

Chris DiBona and Leslie Hawthorn at FISL

Tuesday, June 23, 2009 at 6:06 PM

If you are in South America this week and you have been wanting to find out more about Google's Open Source activities or the future of Open Source and its communities, Chris DiBona and Leslie Hawthorn from the Google Open Source Programs Office will be speaking at the tenth annual Fórum Internacional Software Livre (FISL) in Porto Alegre, Brazil.

Leslie will be presenting tomorrow, June 24th, 1 PM local time about Google Summer of Code™, and she'll be staying afterward to meetup with Google Summer of Code participants who are in the area. On June 26th at 2 PM, Leslie will be presenting "Community Management Basics" and how to make FLOSS projects and communities welcoming for new contributors.

Chris' first talk, "Open Source: Then, Now and Tomorrow" takes place on June 26th at 3 PM local time. In addition to the past, present, and future of FOSS, Chris will discuss how Google uses and releases Open Source software. In Chris's second talk, "An Introduction to Android" at 5 PM on the 26th, he will explain the ideas and structure behind the Open Source mobile operating system Android.

All four talks are great opportunities to learn more about Google and Open Source. Come ask questions and get to know members of the Open Source Team!

by Ellen Ko, Open Source Team

Sojourning at SouthEast LinuxFest

Tuesday, June 9, 2009 at 8:59 AM



Fresh on the heels from speaking at Google Developer Day in Beijing and Tokyo, Chris DiBona will be heading to the SouthEast LinuxFest. This new conference promises to provide a welcoming environment for novice users of Open Source software, with a little something for everyone with talks on data warehousing, education and philanthropy. Should you find yourself in or around Clemson, South Carolina this Saturday, June 13th, please stop by to hear Chris discuss "Open Source: Then, Now and Tomorrow." As much as the Open Source Team enjoys connecting with Open Source developers and users worldwide, Chris is particularly looking forward attending this grassroots, community driven event and sharing his thoughts in this intimate setting.

We hope to see you there and, if you're attending, please do come by to say hello!

Introducing Android Scripting Environment

Monday, June 8, 2009 at 12:52 PM

The Android Scripting Environment (ASE) brings scripting languages to Android by allowing you to edit and execute scripts and interactive interpreters directly on the Android device. These scripts have access to many of the APIs available to full-fledged Android applications, but with a greatly simplified interface that makes it easy to:

  • Handle intents
  • Start activities
  • Make phone calls
  • Send text messages
  • Scan bar codes
  • Poll location and sensor data
  • Use text-to-speech (TTS)
  • And more

Scripts can be run interactively in a terminal, started as a long running service, or started via Locale. Python, Lua and BeanShell are currently supported, and we're planning to add Ruby and JavaScript support, as well.


Scripts can be edited directly on the phone.


The script manager displays available scripts.




Scripts can be launched interactively or as background services.



Interactive terminals can be started for interpreters that support it.



Scripts can use the Android UI to get user input.

You may ask, why write scripts instead of real Android applications? Admittedly, Android's development environment makes life pretty easy, but you're tied to a computer to do your work. ASE lets you develop on the device itself using high-level scripting languages to try out your idea now, in the situation where you need it, quickly. Have a look at the following example Lua script to see for yourself:

--Placing the phone face down will disable the ringer. Turning it face up again will enable
--the ringer.
require "android"
android.startSensing()
android.sleep(1) --Give the sensors a moment to come online.
silent = false
while true do
s = android.readSensors()
facedown = s.result and s.result.zforce and s.result.zforce > 9
if facedown and not silent then
android.vibrate() --A short vibration to indicate we're in silent mode.
android.setRingerSilent(true)
silent = true
elseif not facedown and silent then
android.setRingerSilent(false)
silent = false
end
android.sleep(1)
end

Here's another useful script, this time in Python.

"""Say chat messages aloud as they are received."""

import android, xmpp

_SERVER = 'talk.google.com', 5223

class SayChat(object):
def __init__(self):
self.droid = android.Android()
username = self.droid.getInput('Username')['result']
password = self.droid.getInput('Password')['result']
jid = xmpp.protocol.JID(username)
self.client = xmpp.Client(jid.getDomain(), debug=[])
self.client.connect(server=_SERVER)
self.client.RegisterHandler('message', self.message_cb)
if not self.client:
print 'Connection failed!'
return
auth = self.client.auth(jid.getNode(), password, 'botty')
if not auth:
print 'Authentication failed!'
return
self.client.sendInitPresence()

def message_cb(self, session, message):
jid = xmpp.protocol.JID(message.getFrom())
username = jid.getNode()
text = message.getBody()
self.droid.speak('%s says %s' % (username, text))

def run(self):
try:
while True:
self.client.Process(1)
except KeyboardInterrupt:
pass

saychat = SayChat()
saychat.run()


These scripts demonstrates several of the available APIs available for both Lua and Python. It is intended to be run as a service and silences the ringer when the phone is placed face down. For some scripting languages, like BeanShell, it's possible to access Android's Java API directly. To simplify things, ASE provides the AndroidFacade class. For other languages, like Python and Lua, the API is made available via JSON RPC calls to a proxy. Naturally this means that only the part of the API which has been wrapped by the AndroidFacade and AndroidProxy are available to cross-compiled interpreters like Python and Lua. Thankfully, both AndroidFacade and AndroidProxy are simple to extend.

If you'd like to give ASE a try, it's not yet published to the Market, but will be soon. You can download the latest APK from our project page. Some sample scripts and documentation are also included there to help you get started. We always love to hear what you think, so please send us feedback or ask your questions in the ASE discussion group.

Getting Started in Free and Open Source

Friday, June 5, 2009 at 4:27 PM

The Open Source Programs Office's Cat Allman and Leslie Hawthorn will be presenting at a brand new conference, Open Source Bridge, in two weeks. Their talk, "Getting started in Free and Open Source" will be held on Wednesday, June 17 from 11:20am – 12:05 PM. Leslie and Cat will share their wisdom with Open Source newbies and those looking to attract or retain them. Their talk will cover the basics of:
  • Why you might want to get involved
  • What you can get out of participating
  • What you can contribute besides code
  • How to choose a project
  • How to get started
  • Etiquette of lists and other communication
  • Dos and don’ts of joining a community
The conference is held in Portland, Oregon and runs from June 17th to the 19th. It features over 80 talks in five tracks of Open Source topics and a 24-hour hacker lounge for code sprints. One of the most exciting aspects of Open Source Bridge is that it is entirely volunteer-run, and because this conference brings together developers from all different types of Open Source projects, the structure is designed to provide developers an opportunity to learn from people they might not connect with at other events.

We hope to see you there!

Improving Freenet's Performance

Friday, May 29, 2009 at 10:46 AM

The Free Network project is the community that creates and maintains Freenet, free software that allows you publish and obtain information on the Internet without fear of censorship by means of a decentralized, anonymous network. Since version 0.7 , the software has had built-in support for downloading and uploading large files. These are long-term downloads, which persist between restarts of the node. This support has improved performance and usability, but it has also meant that when lots of downloads are going on at the same time, Freenet uses a lot of memory, takes a long time to complete the startup process, and crashes if you queue too many downloads. By storing the current progress of uploads and downloads in db4o.com's open source object database (= a file on disk) rather than in memory, Freenet's memory usage can be greatly reduced, the end-user doesn't need to worry about running out of memory, we can have an unlimited number of uploads and tens of gigs of downloads, and so on.

To begin at the beginning, Freenet divides all files into 32KB blocks (called CHKs), which are each fetched and decrypted separately. Then we have a layer of redundancy, and various complexities surrounding putting files together and putting in-Freenet websites together, which makes up the client layer. Before the db4o branch, uploads were persistent, but downloads were restarted from scratch after every restart, pulling huge numbers of blocks from the datastore (on-disk cache). Worse, memory usage was rather large if you had any significant number of downloads on the queue. 

The db4o project puts the client layer (persistent downloads and uploads) into a database (db4o). I had initially hoped that this would be a relatively quick project, which shows how much I knew about databases then! We decided to use db4o in a fairly low-level way, specifically to minimize memory usage. We had heard from testimonials that some embedded applications had done this, but unfortunately this is not really the way that db4o is usually used, which caused some complications. Overall, the project took one developer most of a year, the final diff was over 46K lines of code covering 320 files, and went well beyond its original remit, solving many long-standing problems in the process. New architecture was required for optimal performance, including using Bloom filters to identify blocks we are interested in, a queue of database jobs, major refactoring in many areas of the client layer, a new system for handling temporary files, etc.

The effort was well worth it. Our client layer overall has vastly improved and Freenet now
  • starts up quickly

  • resumes work on downloads and uploads almost instantly on startup

  • can have an almost unlimited number of downloads and uploads

  • doesn't need the user to worry about or configure the maximum memory usage

  • doesn't go into limbo with constant 100% CPU usage desperately trying to scrounge a few more bytes

  • can insert DVD-sized files and huge websites (or git/hg repositories) on relatively low end systems

  • uses fewer file handles

This project would not have happened without support from Google's Open Source Programs Office. It will be one of the most important changes in version 0.8  of Freenet when it is released later this year, and current work includes Bloom filter sharing, a new feature that should greatly improve performance both for popular and rare content. Google is also funding that project, watch this space!