Welcome to the new Blue

After about two months of work, I’ve finally completely redesigned the website. I think it sucks a lot less than the previous version, which is always a good thing. It’s written in HTML5 using CSS3. The best part is that I have no idea whether or not it works in Internet Explorer. My traffic is always less than 10% IE, so I’m not going to bend over backwards to support it. Anywhoo, check it out! You’ll also notice that the entire site is now being served over HTTPS.

MacGDBp 1.4: Becoming Asynchronous

In between working on Bugdar 2, Phalanx, and Chromium, I’ve also been working on MacGDBp 1.4.

With the 1.3.0 release, I tried to address a couple of issues with the underlying network communication layer (called SocketWrapper, which was an Obj-C bridge to a BSD socket). The primary issue I tried to address was with Unicode characters; MacGDBp really only knows how to deal with ASCII text, which has proven problematic for international users. I tried to address this with the 1.3 branch by switching how data was stored and processed. But the 1.3.0 release was an utter failure from this perspective. It crashed extremely often, which lead to the 1.3.1 release that reverted the new network-layer changes. (That said, the new features introduced by 1.3 have been well-received.)

But the issues with the network layer ran far deeper than just dealing with Unicode text. MacGDBp uses synchronous communication with the debugger engine. When you issue a command (step in/out/over, get source, get properties, etc.), the command would be sent and then the thread would block until it received a response. This is bad, especially because MacGDBp does not use background threads; all the communication happens on the main UI thread. This can lead to beach balling and a bad user experience. So, why did I do this? Because synchronous communication makes for a dead-simple API. Asynchronous communication requires a lot more bookkeeping and works through callbacks.

So, for 1.4.0, I decided it was time to revisit the debugger backend. Rather than just trying to adjust how data is transmitted (ASCII vs. Unicode), I decided to rewrite from scratch the entire Xdebug communication layer. Rather than using raw BSD sockets, the 1.4 branch uses the CFNetwork API. This yields two huge benefits. The first is that all network activity is asynchronous because socket stream events are scheduled on the run loop. When data is available, a callback is executed and the UI is updated (as opposed to blocking the thread while waiting for a response). This also makes the application more robust, because if the response never comes, it will not lock up the entire UI. Secondly, by using a Foundation-level framework, UTF8 support comes pre-baked via the toll-free bridge to NSString.

I’ve been rewriting the back end in my spare time and it’s finally starting to stabilize. This release is still fairly far out, though. This core network change is also leading to a large refactoring of the entire application to make it more easily unit testable in the future. One of the results of this is creating a new LoggingController to record all network activity. The goal with this “hidden feature” is to make it easier to track down causes of bug reports. The log looks like this; it’s not pretty, but it should get the job done:

I haven’t yet selected the set of feature enhancements that are scheduled for this milestone. Before any new feature work can be started, the 1.4 trunk needs to become as stable as the 1.3 branch. If you have any feature requests, please search for or file new issues and vote them up in the bug tracker.

Tracking Info:
Current SHA1: a7725f8
Version: 1.4.0.40 β
Last Release Build: 2010-03-29 17:51:39

Public Repos and Bug Attributes

Bugdar 2 is now sufficiently far along that I’m comfortable posting the git repository. The Phalanx repository has been public from the beginning (albeit unnanounced) at Github: http://github.com/rsesek/phalanx. Bugdar 2’s repository is now also available there: http://github.com/rsesek/Bugdar2. At the same time, they will be availiable at their canonical home in https://bluestatic.org/git/. The Github mirrors are just easier for most people.

Development of Bugdar 2 is still moving fast. In the past few days I’ve laid down the core of the search system and added support for resubmitting a POST request after login (don’t lose your work if you’re not logged in). There’s two more major milestones before I consider us past the ‘first alpha’ stage: usergroup permissions and bug attributes. What are bug attributes?

In Bugdar 1, there were a set of fields that Bugdar compelled you to use: product, version, status, resolution, priority, severity, and assignment. If you didn’t want to use one of these fields, you couldn’t remove it — you just had to ignore it. There were also custom fields, but the architecture under them was always shaky. By having all these required fields, we were defining a bug as so:

CREATE TABLE bug (
	bugid int unsigned NOT NULL AUTO_INCREMENT
	userid int unsigned NOT NULL
	dateline bigint unsigned NOT NULL
	product int unsigned NOT NULL
	component int unsigned NOT NULL
	version int unsigned NOT NULL
	summary varchar(255) NOT NULL
	priority int unsigned NOT NULL
	severity int unsigned NOT NULL
	status int unsigned NOT NULL
	resolution int unsigned NOT NULL
	assignedto int unsigned NOT NULL
	duplicateof int unsigned NOT NULL
	dependency text NOT NULL
	hidden smallint unsigned NOT NULL
	initialreport int unsigned NOT NULL
	-- [A lot of other cache information here]
);

That’s a lot of data! If a bug isn’t going to use one of those fields, it’s still going to carry around all that weight. In Bugdar 2, this is how we define a bug:

CREATE TABLE bugs (
	bug_id int NOT NULL PRIMARY KEY AUTO_INCREMENT,
	title varchar(250) NOT NULL DEFAULT '',
	reporting_user_id int NOT NULL DEFAULT 0,
	reporting_date int NOT NULL DEFAULT 0,
	hidden boolean NOT NULL DEFAULT 0,
	first_comment_id int NOT NULL DEFAULT 0
);

That’s a huge difference. You’ll notice all the attributes — the metadata — are not stored on the bug itself anymore. Instead, that’s done through a tuple:

CREATE TABLE bug_attributes (
	bug_id int NOT NULL DEFAULT 0,
	attribute_title varchar(50) NOT NULL DEFAULT '',
	value varchar(250),
	PRIMARY KEY (bug_id, attribute_title)
);

The implications of this are that an single (unique), arbitrary attribute can be attached to a bug at the user’s will. The requirement of data is now imposed by the administrator rather than the developer. It also helps to normalize the database schema. So how do attributes work?

Attributes can be defined formally in an administrative panel, or they can be defined ad-hoc while editing bugs. Attributes can be titled to create a field, or they can just be values, which will be represented as tags. The goal with this system is to create a highly flexible data model in which all the stock Bugdar 1 fields could be represented as attributes.

Formally defined attributes can be one of five types: text, boolean, list, date, or user. Text fields are self explanatory. Booleans are interesting because they don’t necessarily have binary value; they have ternary value: true, false, and unset — but that’s more of a semantic detail. Lists are a set of selectable, pre-defined options. The last two types are new since Bugdar 1. Date fields will have a calendar selection widget. Finally, user fields allow the bug to be attached to some user in Bugdar’s system; this is how assignments work.

This is a rendering of how attributes will be arranged on the bug display:

That same information takes up more than half the page in Bugdar 1. That’s all for now.

Long-overdue update

So I know I promised at Thanksgiving an update on the state of things. Well I lied/got busy. Sorry; these things happen. The past few months have been a whirlwind. So here’s a brief personal update before I talk about software.

In May 2009 the Mac port of Chromium (the open source project that Google uses for their Chrome browser) was starting to gear up. I’d been interested in working on a Mac browser project for a while; but Firefox’s code seemed beastly and I never have been a fan of XUL. Safari is great, but it isn’t open source. Enter Chromium: an open-source, truly native, WebKit-based browser. Consider it love at first sight. Fast forward 12 patches and four months later to August when I am nominated and made a committer. My work has all involved bridging the cross-platform C++ model (as in MVC) to Objective-C/Cocoa land. I wrote the Mac the page info window, history menu, cookie manager, font and language settings, download settings, and added favicons throughout various parts of the UI. Now, present day. Last week Google offered me an internship position in Manhattan. I’m thrilled at the opportunity and am excited to start work there in May with such smart, interesting, and talented people. I expect to learn a lot!

So now let’s talk software.

Bugdar

Last time I wrote about the future of Bugdar 2, I posted a link to a list of phases development that would occur. The original plan was to create 1.3 as a stepping stone to 2.0. That line of development has since been abandoned. Instead, I decided to greenfield the project. There were many reasons for doing this; but here are the two big ones:

1. I have a better, more focused vision of what I want Bugdar 2 to be. When writing Bugdar 1, I didn’t have a clear goal of what I wanted — there was no philosophy to the design, I was merely thinking “copy Bugzilla, but do it better”. Overall, I think I was successful with this. But this is not Bugdar 2, at all. Bugdar 2 will be a refined, incredibly flexible system that I am excited to share with you. But because the two projects will be so different, trying to create a large, temporary scaffolding system (version 1.3) was extremely time consuming and kludgey.

2. ISSO. Originally dubbed the “Iris Studios Shared Objects” framework. This thing is ancient. The first version dates back to when I was a rather n00b PHPer. The first commit was from 12 January 2005! Since that time I’ve had a lot more development experience and completed a Minor in Computer Science. A better, more-efficient architecture could be devised.

I wrote an entirely new PHP framework (that will require PHP 5.3+) that is a bare-minimum, no-frills framework. I call this Phalanx and the code is currently available on Github; it is licensed under the GPL version 3. Phalanx is event-driven. Rather than writing another MVC framework, I decided to look at the problem of web frameworks through the lens of functional programming. The problem with MVC on the web is the lack of state; HTTP is not persistent. Functional programming focuses more on inputs and outputs (and the transformations between them), which is how Phalanx works. Each Event class defines a set of inputs that it accepts and the outputs it will return. This also makes unit testing much more clear-cut. I won’t get into further details here, though. That’s for another post.

So before I could start moving again on Bugdar, I had to write this new framework. That took a matter of months (July – January). But I’m now back on Bugdar. Phalanx makes development really fast, and in the past two days I’ve implemented the basic user system, a preliminary Auth2 API, and initial submission and listing of bug reports. I’ve posted a new roadmap for Bugdar 2.

MacGDBp

I’ve been refactoring and cleaning up MacGDBp’s code over the past few months. It’s also been accumulating various fixes for non-critical bugs. I’d also like to re-tackle the socket/network layer for 1.4, as I tried and failed to improve this in 1.3. You can probably expect this update before Spring.

Site Redesign

I’m also working on redesign the Blue Static site. It really hasn’t changed since 2006 (insert the ‘learned a lot’ bit here) and it’s time. I’m not very far on this, yet.

Wow that’s a lot. More in the next few days. UI mocks to come.

Happy Thanksgiving!

Happy Thanksgiving to all you US citizens!

Today I’m releasing a security update to Bugdar, version 1.2.4. This version addresses a possible SQL injection vector in search.php. The details of the report can be found here. Thanks to Adam DiCarlo for disclosing the issue.

Please update to 1.2.4 as soon as you can. The release links can found on the normal Bugdar webpage. Alternatively, here are the direct ZIP and Tar links.

I apologize for the lack of posts, but I’d rather be coding rather than talking about coding ;). I’ll post an update on the state of Bugdar 2.0 in a few days.