Archive for the ‘Uncategorized’ category

PHP File Uploads, FPM, Systemd, and SELinux

Last year I migrated this (and other sites) off a DreamHost VPS, which is a managed VPS product that does not offer much control over server configuration, to an unmanaged VPS at Vultr running Rocky Linux.

The server runs nginx and PHP using the FastCGI Process Manager. Rocky Linux, being a RedHat/CentOS-based distribution, also runs with SELinux enabled. SELinux greatly improves the security architecture of the system, but it does sometimes create obscure problems that take some time to debug. This post will discuss one such problems.

The issue was that PHP file uploads, such as in the bug tracker, were failing to upload, but there were no obvious error messages in the PHP error log.

When an <input type="file"> is used to submit a HTTP POST request, PHP reads the uploaded stream into a temporary file, and it stores the name of the temporary file in $_FILES['userfile']['tmp_name']. Unless the upload_tmp_dir php.ini variable is specified, the files will be put into the default temporary directory, which in this case was /tmp.

In addition, the systemd unit file for the php-fpm service specifies PrivateTmp=true. This option “sets up a new file system namespace for the executed processes and mounts private /tmp/ and /var/tmp/ directories inside it that are not shared by processes outside of the namespace”. In practice, that means that php-fpm runs with /tmp mounted from something like /tmp/systemd-private-026f628ae5ca469eb0213f719ca482-php-fpm.service-blDUvc outside of the namespace.

The issue is that php-fpm does not have permission to write this location because of SELinux. To fix this, I needed to adjust the system SELinux policy to permit php-fpm to write to that location. To do this, I issued the following command:

sudo semanage fcontext -a '(/var)?/tmp/systemd-private-(.+)-php-fpm\.service-(.*)/tmp(/.*)?' -t httpd_tmp_t

… which specifies a regular expression to the PrivateTmp path to be treated as the httpd_tmp_t SELinux label. That grants php-fpm the authority to write files to that directory, which is mounted at /tmp for the process.

After adjusting the policy, PHP file uploads worked like normal. The main confusing bit was that there were no error messages printed to SELinux log nor the PHP log.

Bending PHP to Your Will

One of the best features of Objective-C is protocols. These are akin to PHP and Java interfaces, but have one huge advantage: you can mark certain methods as required or optional. This is useful in implementing the delegate pattern because if you do not care about a certain method, you can simply not implement that method. In PHP, this is not possible with interfaces: you must implement all the methods of an interface in order to implement the interface. While this is fine and makes sense in most cases, it’s not always convenient.

So I set off to find a way to create protocols in PHP. The language is fairly dynamic, so I was able to come up with a solution called WeakInterface. Before jumping into the implementation details, let’s see how it works in practice:

First you define your interface as you would normally:

interface AllOptionalInterface {
  public function DoSomething();
  public function DoSomething1(Closure $arg);
}

The class that implements this only cares about the DoSomething method, so it implements it:

class AllOptionalImpl {
  public function DoSomething() {
    echo 'I did something!';
  }
}

Note that this class isn’t said to implement this interface because it does not implement all the methods. Now in the code that is going to call this interface, you do the following:

$delegate = new WeakInterface('AllOptionalInterface');
$delegate->Bind(new AllOptionalImpl);
$delegate->DoSomething();
$delegate->DoSomething1(function() { echo 'Wow!'; });

This code will output “I did something!”. Let’s go line-by-line. First, $delegate is being created as an instance of WeakInterface, whose constructor takes the name of the interface to “implement.” Then the interface is bound to an instance of the implementation. And then finally two methods are called on the interface, and the one with an implementation is actually called. Calling DoSomething1() is a no-op.

I mentioned above that in Objective-C you can mark some methods as required, too. And that’s done with a docstring when using WeakInterfaces:

interface OneRequiredInterface {
  public function DoSomething();
  /** @required */
  public function DoAnything();
}

If you failed to implement DoAnything(), the call to WeakInterface::Bind() would throw an exception. But what if the number and type hints of the arguments of an implementation do not match that of the interface? WeakInterface has that base covered, too. Bind() will also check the parameters of all method implementations to make sure that the signatures match the interface’s.

That’s WeakInterface in a nutshell, but how does it actually work? It’s easiest to start with a callstack. So when $delegate->DoSomething() is called, this is how WeakInterface actually invokes the implementation:

#	Time	Memory	Function	Location
1	0.0033	975328	hoplite\base\WeakInterface->DoSomething( )	../example.php:14
2	0.0033	975856	hoplite\base\WeakInterface->__call( )	../weak_interface.php:0
3	0.0033	975856	hoplite\base\internal\MethodImp->Invoke( )	../weak_interface.php:84
4	0.0033	976568	ReflectionMethod->invokeArgs( )	../weak_interface.php:145
5	0.0033	976608	AllOptionalImpl->DoSomething( )	../example.php:0

WeakInterface implements the magic PHP method __call(), which is invoked whenever a method call is performed and no method is found by the runtime. It captures this and then forwards the invocation on to an implementation helper MethodImp, which then uses reflection to finally invoke the proper implementation.

Going deeper, when a WeakInterface is constructed, the interface passed to it is reflected and the method list is walked. For each method, a MethodImp is created. When the WeakInterface is bound, that set of MethodImps is looped over, checking that any required methods are implemented and that any implementations match the interface’s method signature. At call time, the method name is looked up in the MethodImp table and the arguments are forwarded to the actual implementation.

One note on performance: due to the number of intermediate method calls, using WeakInterface is roughly four times slower than a plain method call. For most applications, this should not be significant at all; but for performance-critical code, you should steer away from WeakInterface.

The code is available here along with a unit test.

Downloads Work Again

Due to a server misconfiguration issue, the site couldn’t connect to its database. As a result, downloads stopped working since yesterday’s rollout. This has now been fixed. Sorry for any inconvenience.

Welcome to the new Blue

After about two months of work, I’ve finally completely redesigned the website. I think it sucks a lot less than the previous version, which is always a good thing. It’s written in HTML5 using CSS3. The best part is that I have no idea whether or not it works in Internet Explorer. My traffic is always less than 10% IE, so I’m not going to bend over backwards to support it. Anywhoo, check it out! You’ll also notice that the entire site is now being served over HTTPS.

Toolchain and Gitcrement

I’ve posted today a git repository called Toolchain. This is a collection of scripts that I use that others may find helpful. You can find the repository here. In this post, I’m going to talk about one of the items in Toolchain called Gitcrement.

Gitcrement is a set of two scripts that solves the problem of creating build numbers when using git for source control. Unlike Subversion, where the repository revision is the perfect choice for a build number, git has no such luxury because it uses SHA1 hashes for identifiers.

This is where Gitcrement comes in. Gitcrement is a simple interface to a database that contains a sequential ID number (the build number), a username, the current date/time, and a git SHA1 hash.

The gitcrement.py has four commands: init, next, list, and current.

To create a new Gitcrement database, go into any git repository root and type “gitcrement init“, which will create a new database file called .gitcrement.

When you want to create a new build number, simply type “gitcrement next” to record the current user (`whoami`), date and time, and whatever SHA1 hash `git info .` would reveal.

To see the current build number type “gitcrement current” and to get a list of all the build numbers type “gitcrement list“.

That’s all there is to it. To make this system useful, however, it needs a script that can be run in Xcode to automatically call “next” and save the value as CFBundleVersion in Info.plist. So, there’s another Python script called xcbuild.py, which you can use as a Custom Shell Script Build Phase. This will advance the Gitcrement number every time you build the “Release” target in Xcode.

Gitcrement requires Python 3.0, which is not installed by default in Leopard. You can find it here or through MacPorts or fink.

Also, you may need to edit the installation paths in the tops of both files to point to where your copy of “git” is (in gitcrement.py) and where you install the “gitcrement” script (in xcbuild.py).

You can download Gitcrement here:

Link in Gitweb

Source TGZ

Chromium for OS X

Mac users interested in Google Chrome can now try out the latest pre-alpha development version. I have compiled the source tree and the binary can be found here. I’ll be rebuilding and posting the build roughly once a week.

As it says on the page, these builds are unstable and should only be used to see how development is progressing. I will not support these builds and (I’m assuming) neither will the Google team. That being said, enjoy.

Why Go Open Source?

People sometimes ask me why I don’t charge for the software I write. I explain to them that, to me, it’s not about the money. But that’s only part of the story. After thinking about it there are a few reasons why I release everything as open source.

1. You’re Giving Back to the Community

This is what the entire FLOSS community and its mentality is about. You have this great software and you want to give back to the greater community by making it open source. Doing so means that all developers can see how you’ve chosen to implement certain things, they can borrow code, and they can help improve software by submitting patches. For me, I learn a lot of programming languages and techniques by dissecting open source software, and I feel it’s only fair for me to give back to the community as much as it has given me.

2. You Don’t Worry About Business

I’ve been reading some articles about indie development and I watched all of the [C41 videos]1. From doing so, I’ve realized that starting an independent software shop is a lot of work. You have to get a credit card processor, you have to have a licensing infrastructure, you have to worry about piracy, and (arguably the most difficult) you have to find the right price for your software. We’re software developers, not business people. This is not my idea of a good time and I’d much rather spend time developing rather than doing the aforementioned tasks. Going open source frees you from all of that.

3. People Can’t Have Expectations

That is sort of a half truth. People will always have expectations about software, regardless of whether or not they pay for it. But when you develop open source software, you’re essentially a volunteer. People get what they pay for. I try to do my best in creating software and supporting it, but because it’s open source, there’s no expectation that I have to. As a full-time college student, I don’t have a lot of free time — I develop when I can. If I had paying customers, there’d be an expectation for future versions on a reliable time scale, but I’m not under that pressure. Additionally, I do not have to support this software with immediate email responses, blog follow-up comments, or forum posts. I do so because I like doing so. If this were a business, people would have high expectations for support, but because I do open source work, if I deliver good support, it’s only a bonus. If I don’t, they really can’t complain because they don’t pay for anything.

Pipeline Update

It’s been a while since I discussed pipelines, so I’m going to talk in this post about the state of various projects, as well as some talk on new projects!

COMPLETED PROJECTS:

MacGDBp

Released this summer, MacGDBp has been a tremendous success. This week version 1.1.2 was released, which address two bugs, one of which was a crasher. I’ve got big plans for version 1.2 and 1.3, but these will have to wait for other projects to be completed.

CMS

In my last pipeline update, I mentioned the CMS I was building for my university’s TV station. This project launched only two weeks ago, so the past month and a half I’ve been working feverishly to get this done. It’s now complete and so this will no longer be affecting development.

CURRENT PROJECTS:

Bugdar 2

The first milestone in Bugdar 2’s development phases was completed a few weeks ago. Since then, not much has happened because all of my effort went to deploying the aforementioned CMS. However, work will be resuming on phase 2 and development will be continuing. This week I went through the Blue Static bug tracker and triaged a bunch of bugs that had been left unresolved.

NewzGrab

I’ve started working on this project in the past few days. As stated in the previous pipeline posts, this is a Usenet downloader for NZB files. However, I’m writing this as a cross-platform application. Most of the code is being written using C++ and Boost, which allows it to be platform-portable. I intend to then create native interfaces for various platforms (initially, only the Mac one will be created).

MyWishlist

This project is mostly complete. A few interfaces need to be updated and created, but on the whole this is largely done. I haven’t actively worked on this in a few months, and I consider it to be one of my lower priorities.

NEW PROJECTS:

iD3

I realized the other day that the Mac lacks a good, free ID3 tag editor. I quickly found id3lib and wrote a simple Mac interface for it. Version 1.0 will be released in a few days, but will be very minimal. I intend to add more features (and more tags to edit) in the future, but the necessary fields will all be present in v1.0.

“Commander”

This is the project I’m most excited about. It’ll be a unique take on launchers (e.g. QuickSilver, Butler, LaunchBar) that is for pro-users only. I won’t be going into the details until I have something to show, but this will be one awesome app!

Scrabbalize 2.0

I’m pleased to announce the immediate release of Scrabbalize 2.0. This version adds wildcard support. Merely add a ‘?’ to your tile set and it will match any letter:

And as you can see, the new version has a completely redesigned interface that looks fantastic in 10.5. Finally, as an extra treat, I have made it 10.4-friendly. So you can now run Scrabbalize on Tiger!

Enjoy.

The Road to MacGDBp, Part I

Inmy last pipeline update, I said that MacGDBp would be out by May. Unfortunately, I’ve run into two bugs that I’m not quite sure how to work around. One of them involves NSXMLDocument raising assert() messages for what appears to be valid XML. Weird, right? So instead, I’m going to write an article about MacGDBp in hopes of whetting your appetite for it until it’s done.

History

The idea for MacGDBp has been in my head for a long time. PHP debuggers are far superior to var_dump() and print_f() debugging statements. Unfortunately, only the Zend IDE was available — but it cost money and was ugly (i.e., not Mac-esque). Next came Eclipse with the web tool kit. I had a heck of a time getting the debugger to work and ultimately gave up. Additionally, it too was ugly. So I put up with my combination of BBEdit (and later TextMate) and var_dump()/print_f().

Fast forward a few years to the introduction of the Xdebug extension. This extension is absolutely fantastic, if only for the stack traces in errors and uncaught exceptions. But it also has a remote debugging function. For about a month, I used the packaged command-line demonstration tool and read XML to glean debugging information. Then at the end of July/beginning of August 2007, I made the decision to write a native Mac application for the Xdebug remote debugger.

Evolution of Design

It’s rare for me to design an application without first sketching out the interface on paper. But because this was such a straight forward project, I decided to wing it. The second commit in the git repository was “Laying out an unhooked basic interface for the system.” This was the first revision.

Yes, ugly. But it improved after some tinkering:

Then I realized that I was trying to re-invent something that had already been well-designed. So with the next iteration, I mocked up Xcode’s debugging window.

After designing a couple of toolbar icons, we get to the final design. This is how the application will look when launched and unconnected to Xdebug:

And there you have the final design. In the next couple of days I’ll talk about how I designed the icon in an effort to stall while I squash the last few bugs.

A Note about System Requirements

MacGDBp will require 10.5. And this is not because I don’t want 10.4 support, but MacGDBp relies on a bug that was fixed in 10.5. In Mac OS X, there is a class called NSTreeController that I use for the variable list. But under 10.4, parts of this class were not properly exposed and were very buggy. I found a work-around, but it relied on using private, undocumented API calls. In 10.5, Apple fixed the bug and exposed the necessary parts.

« Older Entries