Planet Bazaar

Elliot Murphy: MySQL, “what if”, and the drizzle project

Tue, 22 Jul 2008 19:43:52 +0000

Looks like drizzle is announced now. I’ve spent a bit of time after work and on lunch breaks helping out here and there, and I’m excited about working on a database project again. Why am I working on the project? Average time from when I write a patch to when it goes into the tree has been measured in minutes, not in hours/days/weeks/months. Yes, I’m running the test suite first. Yes, I’m getting another person to review the code first. This is an example of how adding people to a project can slow it down, and how getting out of the way of the engineers can have amazing results. We set up bug tracker, code hosting, team organization, package build system, mailing list, IRC channel, and more in a matter of minutes, and it has been amazing to see how fast the code is coming in from all over the world. There was truly a pent-up demand for somewhere to be able to freely work on ideas.

If you haven’t read Ian’s paper about the community-agile process, you should. You can see many of the ideas there in how the drizzle project is run. There is lots of work to do, and I think it’s going to be great fun to see how far we can push drizzle. And the code is safely in a FLOSS distributed version control system that I have mirrors of, so nobody can put the code behind a corporate firewall and seal it off - it’s alive and growing and unstoppable. If anyone wants to mirror the code, that is fine and I’ll help you do it.

Sometimes people look at databases as boring, as a solved problem. I’m here to tell you that building interesting applications that have any kind of persistent state is not a solved problem, there are a lot more fantastic ideas to try, and some ideas that have only recently become practical. Check out the code and put up a branch with some ideas of your own!

We don’t have packages for ubuntu yet, but we will be putting up a PPA soon. As you can imagine, there is still a fair amount of work involved in finishing the renaming and making packages that don’t conflict with existing mysql installs.

John Arbash Meinel: This Week in Bazaar

noreply@blogger.com (jam) — Thu, 17 Jul 2008 19:42:00 +0000

Welcome back to the terrarium of the Bazaar distributed version control system. Written by co-authors John Arbash Meinel, one of the primary developers on Bazaar, and Paul Hummer, who works on integrating Bazaar into Launchpad as he refines his plans for world domination from his shiny new lair.

Bazaar 1.6b3 released

The next beta release of Bazaar has just been cut, and is available at your local PPA:
https://launchpad.net/~bzr/+archive

The Windows installers should be available later today. This release provides lots of the shiny things that we've been talking about, like Stacked Branches, Real Weave Merge, more hooks for server-side operation, and lots of bug fixes and general polishing. The full UI for using stacked branches still needs a little bit of polishing, so the feature is not enabled by default. The functionality is all there, and if you are interested, we'd love to hear from you (kudos and complaints are equally welcome).

New updates to Gnome Bazaar Playground

Coming back from a very productive trip to Guadec, Tim Penhey has been overseeing some customizations to the Bazaar Playground for Gnome. All of the branches created at the local server in Turkey for Guadec have been added to the public playground. The Loggerhead installation has received some TLC by way of customizations to the UI. Accerciser's playground page is a good demonstration af the UI changes that have been made. The playground is actively being used by applications such as Brasero, jhbuild, Metacity and more.

One of the fun results of meeting with people at Guadec, is that it showed ways to improve Loggerhead when dealing with lots of projects and lots of branches. Work is continuing to make customizing Loggerhead's look-and-feel easier, and providing better tools for creating these "Bazaar Playgrounds" to use in evaluating Bazaar. The Bazaar developers are committed with making tools easier to use, and making the process as simple and powerful as possible.

Up and Coming Repository Format Updates

Robert Collins has been hard at work to refine how Bazaar stores its history information. We all like to have deep context, but we don't like to have to pay the penalty of downloading all of that context. Because Bazaar has a flexible repository structure, Robert has been able to play with changing the on-disk structure without major surgery to the rest of the code.

First is a change to how indexes are written, switching from a bisectable list to a btree structure. This paged structure allows us to compress the indexes, making them smaller, and faster to process remotely. It also reduces the number of lookups to find a key. (On average, a bisect search is log₂N, while the btree is closer to log₁₀₀N.) At the moment, he is testing this with a shared repository containing all of the projects available from in the Ubuntu apt repositories. This weighs in at around 13k branches, and somewhere around 20GB of disk space used.

Second is an update to how texts are stored. At the moment we use a simple format which places fulltexts periodically, and then stores deltas against those fulltexts. It has served us rather well, but can be improved upon. With his Group compress work, we can see a savings of as much as 2x-3x. Further, the data is stored such that you can do simple linear reads to get the base fulltext and all deltas necessary to generate a given fulltext. This reduces the pressure on indices, as you don't have to search for base texts. (Instead you just store a pointer to the start, and give the total length that needs to be read.)

These are still in development phase, but a format that uses them will likely appear in the next release (bzr 1.7).

Community Agile

Ian Clatworthy has recently released a wonderful document describing the workflow we (generally) use at Canonical. It describes how basic practices are similar to, and different from, other systems like Agile. The biggest (IMO) being a recognition that the community surrounding your project is one of the strongest and most important pieces. This has always been true in software development, but it has traditionally been somewhat hidden. Open Source has exposed just how powerful the community can be. For people interested in how software can be developed, rather than just what, I certainly recommend it.

Jelmer Vernooij: Oxegen '08

nospam@example.com (Jelmer Vernooij) — Wed, 16 Jul 2008 13:28:07 +0000

Oxegen was a bit expensive but awesome. Irish people rock.

update: Wilmer has details.

Elliot Murphy: Cool chair

Wed, 16 Jul 2008 03:55:52 +0000

Cool chair

Originally uploaded by Elliot Murphy

–
Elliot Murphy

Tim Penhey: Bazaar has the model right

noreply@blogger.com (Tim Penhey (thumper)) — Sat, 12 Jul 2008 00:42:41 +0000

Some people in the GNOME community have suggested that if Bazaar has nice usability, then GNOME can just use Git on the back-end, and Bazaar lovers can just use the Git back-end via Bazaar. It's true that Bazaar could support this — an experimental plug-in exists to do this right now. But this suggestion betrays several wrong assumptions.

People assume Git and Bazaar are the same. They're not. People assume that if Git and Bazaar have technical differences, then Git must have it right.

The problem with these assumptions is that usability begins at the ground level. Bazaar started with a focus on usability. Git began with a focus on speed. The data models of both Bazaar and Git reflect their initial focus. But Bazaar's model can also be fast. In fact, the Bazaar developers are currently optimising a number of key operations for speed.

Data retrieval

Git and Bazaar are both key/value mapping systems. When bytes are needed, they are requested with that key.

The big difference is that Git's keys are also the hashes of the bytes. This is why it's called a content-addressable file system. This allows git to offer a guarantee that if the value hashes to the key, it has not been modified, whether deliberately or by accident. The Bazaar team considered adopting this approach, but decided it was too constricting. Bazaar uses UUIDs instead.

Authenticating revisions

For detecting malicious modification of revisions, Git uses its cryptographic hashes.

Bazaar uses revision-signing. All revisions can be PGP-signed. No signed revision can be forged. And the hashed representation can easily be generated and passed around to ensure that exactly the same content is used.

If SHA-1 is broken, both Bazaar and Git will lose their ability to detect malicious modification. But since Bazaar uses UUIDs to identify revisions, users can re-sign their old revisions with whatever method proves to be secure. Changing the hash used by Git would make it incompatible with all existing repositories.

Data Integrity and Serialization formats

Bazaar stores hashes of every value, so it equally capable of detecting accidental modification. It can be useful to have different representations of a tree in different repositories. For example, when Git lists files, it divides this data by directory. This is a good approach, but not necessarily the best approach. An alternative approach would be to use a radix tree. This would ensure that Git performed quickly even if users put unreasonable numbers of files in a single directory. But Git's keys are hashes, upgrading Git's format to use radix trees would change the keys, which means that people could not use the commit-id from one repository to refer to the same tree in an other form.

Bazaar doesn't assume it has the perfect format. It provides an upgrade path, and does't change the commit-id of a revision if you change your format. What's more, Bazaar can even reference data it has never seen. This allows partial imports from other VCSes to be fully compatible with more complete imports. And if a VCS provides UUIDs (content hashes certainly qualify as UUIDs), Bazaar can refer to those UUIDs directly.

File and directory representation

Git refers to files by path. It makes no attempt to track renames in its data store.

Bazaar has an inode abstraction; files and directories both have ids. When a file is renamed, its id stays the same. Bazaar's core code refers to files by their id, so merging a renamed file requires no special effort.

Git's approach means that users are warned not to rename files while changing their content. But when files are renamed, those files that refer to the renamed files must have their contents changed as well. For example, if you rename foo.h and foo.c to bar.h and bar.c, you should update the contents of bar.c, or else you will break the build. With Bazaar, users can do whatever they want, and the VCS just works. While Git must always use heuristics to deduce renames, Bazaar does not have to. Of course, it can if it wants to. This is an example of why it is important to design a model for usability from the beginning.

Bazaar can import rename data losslessly from foreign VCSes. Some other VCSes support file-ids, and Bazaar can reuse those without change. For VCSes that support renames, but not file-ids, Bazaar's representation is also non-lossy. When data imports are deterministic and non-lossy, it's easy to export them back to their source VCS. Bazaar's Subversion integration is a great example of how this can work.

Choose the back-end with the right model

In any situation it makes sense to use a back-end that stores the richer dataset. It makes more sense to have a front end client that doesn't use all the functionality or data representation of the back-end than it does to have a richer client that isn't able to store the required information as the back-end is not able to represent it.

If a single back-end storage is going to be used, it makes more sense to use a Bazaar back-end as Bazaar is able to represent everything that Git does, but the reverse is not true.

Conclusion

The Bazaar developers focused on usability, which requires having a model that supports usability. Bazaar has improved its model to increase the usability of the system. We believe that Bazaar has the right model.

co-written by Aaron Bentley and Tim Penhey

Robert Collins: 11 Jul 2008

Fri, 11 Jul 2008 13:12:22 +0000

What the git vs bzr discussion is about IMO is usability. The following blog post about DTrace on linux talks about the same issue, and I'd like to use Bryan's words:

"Over and over again, we made architectural and technical design decisions that would yield an instrumentation framework that would be not just safe, powerful and flexible, but also usable. The subtle bit here is that many of those decisions were not at the surface of the system (where the discussion on the Linux list seems to be currently mired), but in its guts."

"Over and over again, we made architectural and technical design decisions that would yield a Distributed VCS be not just safe, powerful and flexible, but also usable. The subtle bit here is that many of those decisions were not at the surface of the system (where the discussions going on at the moment seem to be currently mired), but in its guts."

Tim Penhey: Re: Interesting (rough) statistics at the GNOME distributed RCS BOF

noreply@blogger.com (Tim Penhey (thumper)) — Thu, 10 Jul 2008 23:17:41 +0000

J5 mentioned in his post his interpretation of the number of users for GIT, Bazaar and Hg (Mercurial). He also finishes with "Converse amongst yourselves".

I guess I should first point out that I am a Bazaar user, and that I work for Canonical. I felt somewhat enraged at the post from J5, and have spent some time trying to work out some response.

John Carr mentioned that 83% of statistics are made up on the spot, and that cannot be more true here. I had been waiting for someone else to post the numbers that they saw at the BOF, but so far I have not seen one.

Here is my take on it.

Yes there were more GIT users than Bazaar users at the BOF, but the numbers were more like 50% of the audience were GIT users, and about 40% were Bazaar users. Someone piped up and said "What about Mercurial?" and so the question was asked, and there were about five or six people. There was an overlap of the GIT and Bazaar groups, and there was by far the larger majority of the audience that had not used any DVCS.

What conclusions can we draw from this? Not much. Many people attending the pre-conference work for larger companies, like Red Hat, Novell, and Nokia, and many of those people work on some hard core linux stuff, many of which have chosen GIT. Many have chosen GIT because that is what the linux kernel is using. Is that a good reason to chose a DVCS? I don't feel that we can really answer that question as I am sure there are strong advocates for both sides.

An interesting question is "Which DVCS is easier for the casual contributor to use?" Surely one of the reasons that a project chooses a DVCS is to allow for more community contributions in an easy to merge way that has a clear contribution history. Bazaar just works. It works for the hard-core developers, but is also easy for those soft-core (?).

From the people I talk to, and I've tried to talk to many here, is that of those that use Bazaar it just works. Bazaar doesn't get in your way of developing the software that you are working on. It is just a tool that works.

One final point. The questions were "Who uses <insert DVCS>?", not "Who likes/loves using <insert DVCS>?".

John Arbash Meinel: This Week in Bazaar

noreply@blogger.com (jam) — Thu, 10 Jul 2008 20:05:00 +0000

Here we are again, bringing you the gossip and dirty secrets in the development world of the Bazaar distributed version control system. In this, the 10th week, the series is now under new management, with co-authors John Arbash Meinel, one of the primary developers on Bazaar, and Paul Hummer, who works on integrating Bazaar into Launchpad.

Bundle Buggy

Aaron Bentley has once again been improving his wonderful Bundle Buggy. He just introduced support for multiple projects using a single instance of Bundle Buggy. There are now 5 Bazaar projects using the main bundle buggy instance. (Bazaar, bzr-gtk, Bundle Buggy itself, Bzrtools, and PQM.) Of course, Daniel Watkins has made excellent use of his time, and has managed to crank out lots of updates for PQM. At this point it is code clean up, reducing the dependencies making it easier to set up and install.

Bazaar playground for Gnome

Originally, John Carr set up Bazaar mirrors of all the Gnome modules, which people could then use as a starting point for publishing code and collaborating. This week, the Bazaar playground for gnome was created so that any Gnome developers could be involved in pushing, branching, and sharing code through bazaar. This new server runs Loggerhead for viewing the code committed to these Bazaar branches. Damned Lies is also set up on the playground. This server was also reproduced locally at GUADEC because of the flaky internet connection at the conference, and all those local branches will be moved to the playground shortly.

Weave merging and handling "interesting" history

One of the great things about having a large project like MySQL using your software is that they push and stretch you in ways that you haven't necessarily encountered before. Specifically, their branch workflow looks a bit like a pile of spaghetti. With several long-term maintenance branches, team branches based off of that, and individual developer branches based off of that. Patches have a tendency to travel in unexpected ways (you may go user => team => release 1 => release 2, or you might go release 1 => team => team-2 => release 2, etc). They also are very fond of 'null merging' patches that aren't relevant to the next release. They merge the change and revert the text changes and commit.

Bazaar supports all of this, but it exposes weaknesses in simple 3-way merge logic. Because patches don't flow in anything considered orderly, you don't have the opportunity to select a "clean" base very often. Bazaar has long had an option for doing a "--weave" merge. It didn't receive much attention for a while, and had become rather slow. It turned out to be a good fit for MySQL's workflow, so John has spent a bit of time recently to make the functionality efficient and correct in some specific edge cases. Expect the improvements to show up in the next release.

Robert Collins: 10 Jul 2008

Thu, 10 Jul 2008 06:47:48 +0000

I keep running into folk whom I knew of, that use bzr, but I did not know that they use bzr.

Right now there is a lot of discussion going on about DVCS in various projects. While I imagine most bzr users just want to get on with their coding (after all thats what bzr is good at :))... it would be fantastic if you could blog that you use it, and folk at GUADEC wear the T-shirt!

Also, I'm at GUADEC, and I'm extremely happy to answer questions from anyone, bzr user, git user, or even svn user :)

Elliot Murphy: Hiring web, python, and gnome developers

Wed, 09 Jul 2008 15:11:01 +0000

I’m looking for some more unstoppable hackers to work with at Canonical, and doing the usual thing of going through CV/resume submissions. What’s different from when I was doing this last year? YouTube! In the past I’ve been impressed when someone not only had a resume but links to a personal site or profile that highlighted their professional activities in much more detail than a plain resume would - perhaps links to open source contributions that they are proud of, or designs that they’ve done, etc. One resume I just got links to a YouTube video showing a demo of a robot that the applicant built and programmed in C and Python. This definitely got my attention.

Ian Clatworthy: Announcing the Community-Agile project

Tue, 08 Jul 2008 14:47:39 +0000

I’m pleased to announce a new project that aims to extend Agile software development with successful practices used in the Open Source community. The goal is to create and support a process framework that teams and communities, both open source and commercial, can download and customize to meet their needs.

To start the ball rolling, I’m pleased to announce the Manifesto for Community-Agile Software Guidance. Anyone can sign this - simply go to the page and add a comment indicating that you agree! More importantly, a paper explaining the values, principles and practices is available in numerous formats including html and pdf. (I was planning to present this paper at OSCON 2008 but unfortunately can’t travel at this time.)

To get involved, visit https://launchpad.net/~community-agile and join the team!

Jonathan Lange: I Love Data

noreply@blogger.com (jml) — Mon, 07 Jul 2008 19:22:37 +0000

At work, we recently started to graph the time it takes to access a branch in Launchpad via SSH, both on our staging and production servers.

I love this so much. Having this data is liberating. It's like turning on a light in a dark room: suddenly I can go from uncertain, careful, hesitant steps to bold, confident strides.

In fact, the metaphor stretches further. This new light has revealed objects of interest (that is, spikes in the time taken) that I wasn't aware of before. I need to know how long these spikes last, how they correlate with load on the system and so forth. Soon there'll be more graphs, and I'll be able to correlate them and analyze them and suck on their delicious, numerical marrow (the light metaphor long abandoned).

Regardless of whether it's liberating illumination or nourishing, savoury meat, this new graph makes me wonder why I don't chart other things I care about. Having such graphs would help me see the impact of my actions. I could chart things like my bank balance, my waistline or the number of Latin words I learned this week. Then I could answer questions like "do I spend more on Tuesdays?" or "can I afford this iPod/cake/gerund?". This is important, because when I am tempted with a sleek/delicious/perplexing iPod/cake/gerund, I fall back on my own judgment. I think it's time to confess, dear reader, that my own judgment isn't very good. And yet I continue to trust it.

The reason I don't have such graphs is that they are inconvenient to maintain. A graph of branch access times is easy. All you need to do is describe how to get the measurement, and then do a bit of once-off set up. If you want to graph your body weight, you need to get on the scales at a fixed time and then look at the dial and adjust for parallax and then write a number down somewhere and maybe note down whether this is before or after a meal and then take the number and then add it to a spreadsheet. Most online banking sites I've seen are even less convenient than this.

Getting data on life is hard, but for programming it's easy. What things do you care about on your project? Features, user experience, how fast bugs get fixed? Can you quantify these things? Can you make a pretty picture out of it? If so, do it now! Post your answers here, set up something like Cricket and then profit!

P.S. I can't believe I got to the end of this post without saying how awesome the Canonical sysadmins are. Let me correct that now. The Canonical sysadmins are heck of awesome. They could simply walk into Mortor.

Jonathan Lange: Five Years

noreply@blogger.com (jml) — Sat, 05 Jul 2008 22:21:31 +0000

That's not quite how long it took Glyph to write a post explaining what he meant when he told me that xUnit should use the visitor pattern instead of the composite pattern.

I'm going to write a more detailed response later, God willing. I just wanted to flag the post and say thanks to Glyph for finally posting it — the hassling must have worked.

Jonathan Lange: The Economist Index

noreply@blogger.com (jml) — Sat, 05 Jul 2008 22:21:30 +0000

The Economist occasionally publishes a "Big Mac Index" — how much the world's favorite "burger" costs in each country, translated into a common currency.

Why they don't publish an index of how much an issue of the Economist costs? All of my issues have the price of the newspaper in lots of Asia-Pacific currencies. A small sample:

Australia, 9.39855 USD

Cambodia, 6.00 USD

China, 10.43655 USD

Hong Kong, 7.6917 USD

India, 5.0522 USD

Japan, 10.70535 USD

New Zealand, 8.6669 USD

Can anyone tell me why the price varies so much?

Jonathan Lange: Three-ways on Pidgin?

noreply@blogger.com (jml) — Sat, 05 Jul 2008 22:21:29 +0000

Dear Lazyweb,

How do I initiate a three-way chat with Pidgin 2.2.1?

jml

Jonathan Lange: Merging New trunk Features to a Development Branch (redux)

noreply@blogger.com (jml) — Sat, 05 Jul 2008 22:08:02 +0000

Oubiwann has recently posted about the joys of using Combinator to do branch-based development using Subversion. I thought it'd be fun to do the same post, except this time with Bazaar.

Background

You're working on a project called "Project", you have a copy of the mainline branch (i.e. 'trunk') in your src directory.


~$ cd ~/src/Project
~/src/Project$ ls
trunk

You want to implement a new feature, so you branch trunk to work on it:


~/src/Project$ bzr branch trunk viking-feature-836

Bazaar is a version control system, not a PYTHONPATH-managing system, so it doesn't maintain a global list of projects and the branches that are currently active for each project.

Perhaps your company focuses on historical invasions of Britain. You decide to start work on another feature:
lass objects.


~/src/Project$ bzr branch trunk norman-feature-1066

You multi-task for a bit, until you finish 'viking-feature'. You decide to merge 'viking-feature-836' into trunk.


~/src/Project$ cd trunk
~/src/Project/trunk$ bzr merge ../viking-feature-836
~/src/Project/trunk$ bzr ci

At this point, you begin to suspect that Bazaar treats branches as first-class objects. However, at this point, a developer on the obverse side of your continent calls you,

"Where's your viking feature? I need it to invade Britain!"

"I've just put it into trunk. Have you got the latest copy?"

"Yeah, I do, I just pulled from trunk."

"It's in trunk, you fool! ... Oh, wait, gimme a sec."


~/src/Project/trunk$ bzr push
Pushing to bzr+ssh://bzr.example.com/Project/trunk...
~/src/Project/trunk$

"Try now."

Merging

OK, enough background, let's merge.

Say you need some of the changes in trunk in order to finish work on your norman feature. No problems.


~/src/Project$ bzr merge trunk norman-feature-1066
~/src/Project$ bzr ci -m "Merge from trunk."

It's hard not to feel smug at this point.

Wait a second, you also want to look at the experimental branch that a friend is working on:


~/src/Project/norman-feature-1066$ bzr merge bzr+ssh://yourfriend.example.com/branches/sealion-1946
~/src/Project/norman-feature-1066$ bzr diff | less # better double check this one
~/src/Project/norman-feature-1066$ bzr revert # nope, doesn't seem like a good idea

Summary

Bazaar treats branches as first-class objects and treats trunk just like any other branch. Although Combinator is great for branch-based development in Subversion, it is more complex and less flexible than doing branch-based development in Bazaar.

With Combinator, you lose history when you merge in changes from trunk, with Bazaar you don't.

With Combinator, you can only merge in changes from trunk, with Bazaar you can merge from any branch.

With Combinator, merging from trunk leaves a bunch of changed files in the trunk checkout on your system (this has tripped me up more than once). With Bazaar, this doesn't happen.

Jonathan Lange: It's a Gibbon

noreply@blogger.com (jml) — Sat, 05 Jul 2008 22:08:01 +0000

Over the weekend I upgraded my laptop to Ubuntu 7.10 aka "The Gutsy Gibbon".

Obvious improvements include having Pidgin installed and a much superior version of Deskbar. There's also a swanky new panel to actually control display settings (dual monitor and all that jazz). Unfortunately, I can't quite get it to work just yet. My 22" 1680x1058 LCD will just have to sit there, gathering dust :(

In another weirdness, all of my fonts are suddenly much, much bigger. I had to resize everything down to 8 or 9 to make them sane.

My pretty Emacs has also stopped working, but that's to be expected. It's a third-party package that hasn't yet been built for Gutsy.

Other than that, seems to be a fairly routine upgrade. Is there anything exciting that I've missed?

Jonathan Lange: Around like Grover

noreply@blogger.com (jml) — Sat, 05 Jul 2008 22:07:59 +0000

What's up hep-cats?

I've been travelling around the world, weaving in and out of timezones like Jason Bourne navigates facial expressions. Last week's adventure took me to tropical Dunedin — the southiest outpost of Canonical Ltd. (I'm trying to get a mention on Language Log.)

Those few Hobartians who keep reading will notice that Dunedin, well, umm... here's a photo:

Pay careful attention to the hills in the background.

Jonathan Lange: Cracked Mac

noreply@blogger.com (jml) — Sat, 05 Jul 2008 22:07:58 +0000

OK, I've started with this whole photography thing, I might as well run with it.

My Mac cracked:

Jonathan Lange: The most retentive thing I've done

noreply@blogger.com (jml) — Sat, 05 Jul 2008 22:07:57 +0000