Let me start with a little imagination. An analogy, if you will.

Imagine, if you will, that you are on holiday. You want to send a “wish you were here” type message to your friend back home. You have two choices: a cheapish picture postcard, or a more expensive sealed envelope. What do you choose? If you’re like the vast majority of us, you will likely choose the postcard, for both picturesque and cost reasons.

On the other hand, let’s assume you want to send your credit card number and salary details to an insurance agency. Again, do you choose the postcard, or the sealed envelope? I will not insult your intelligence by saying what the “right answer” is. I will however allow someone else to continue this analogy on my behalf, with a slight piece of satire:

the government has announced that it will soon be mandatory to use state-approved envelopes to send all mail. these new envelopes will be entirely transparent when viewed under a federally produced lightbulb, but there is no need to worry about these lamps getting out to bad people, since it is time-tested proof that all government employees are completely honest and lack all self-serving traits present in every other human being. besides, it’s for your own good and protection! and if you have something to write that you don’t want everyone to read, maybe it’s time for that all-important self-examination to reveal your underlying paranoia complex…

(quote taken from a Slashdot comment, written by someone called CrudPuppy)

How does that make you feel? If you honestly sat down and thought about that eventuality, would you be happy about the idea of someone, anyone, being able to read your most private thoughts and words? Is it fair to expect that once it’s written down, “The Government” — whoever that may be at the time — has a right to view it? The fact is, some governments are currently attempting to change their laws to curtail the individual’s right to this sort of privacy, whether you want to believe it or not. For example, the American Wiretap Bill prohibits using emails as evidence without a warrant, but this article shows that the authorities want to read parts of such communcations without a warrant. The new proposal is called the Combating Terrorism Act of 2001, and it considerably extends the surveillance powers of the FBI to such an extent that Senator Orin Hatch said “No reasonable expectation of privacy exists”.

This is an important statement from such a high-profile member of the government of the most powerful country on Earth. Most “civilised” countries (and I use that term semi-ironically in light of the meaning it has developed in U.S. media since Sept. 11 2001) have privacy laws, in which “The Individual”, i.e. me and you and your best friend and the milkman, has a right to privacy, to not disclose any personal or “private” information under normal circumstances. Article 12 of the United Nations Universal Declaration of Human Rights reads as follows:

Article 12.

No one shall be subjected to arbitrary interference with his privacy, family, home or correspondence, nor to attacks upon his honour and reputation. Everyone has the right to the protection of the law against such interference or attacks.

Although this is not enforced as law, it is understood that member countries in the UN follow the spirit of the Declaration. Specifically in this case, correspondence whether written or by telephone is protected from “arbitrary interference”. Practically speaking, the contents of sealed envelopes are protected by law in many countries, except in certain circumstances such as imprisonment. Unfortunately, most of the legislature of these countries, and indeed the populace at large, and the media that feed the populace, are ignorant of the technological counterpart to the “sealed envelope” privacy I have outlined above.[1]

Let me explain. When you send an email, it is like a postcard. Much worse — it is more like a postcard that passes through many different postal services, and can be photocopied at any point. In fact, it is usually photocopied at (at least) two points along the route: the Internet Service Providers at each end of the connection. If either end is in a business, i.e. if you send or receive email at work, then chances are there will be yet another copy somewhere in that computer. And most people are not aware of that fact, that their daily correspondence is routinely copied and stored by entities outside their control.

Encryption

Enter encryption. Encryption is a mechanism whereby the sender “scrambles” the contents of a communcation at source, with the intention of maintaining privacy until the intended recipient decrypts or “unscrambles” the message. The equivalent of a sealed envelope, if you will. The sealed envelope can be photocopied by anyone en route, but the only useful information on the copy will be the address of the recipient. Similarly, an encrypted email would look like the following:

—–BEGIN PGP MESSAGE—–

Version: PGP Software

Comment: Any Plain text Message you like

hQEOAz23QZbhSPHnEAP/fNq8vWQg0qi0VN/L5QVS8R+Jwvzx+oHSxHOAHsepUYZl

msVIVgYY4g4Ptd4numv67W2+y/eW4qhTlInOmHloAZtu239FPjad/mwapfC4z/EA

QoyhsuzaQt6NsDpRvMnMSW5zwAMQBAlt/b/e3CTO6GpxOehFzPiesi1ltX8hU10D

rNCadlmjwEB7xi9oqcmFawRQmhkNAZ2SWYcxOk3yGXlkx+NWdewq3r+VM8KHdhA

yuT75f+Dw58+jplyjFWhITCZREDSt/EH56K/6NaUVtKI0sDMARWbt69BQSYMO3At

3IkpDtSFdJREJRKK8fpa+hBa9pdGaG1nmTNJYuR91FAYjdll1Rbzfu72zLYnRnuJ

21ECE0rHN3dmcv21ZMtPn8Q8mWTRHapavgJqJkA0GRQmmZ7niJMxi8YknyJwyvyG

=kn6s

—–END PGP MESSAGE—–

As you see, it’s a lot of garbled nonsense, and could be copied as often as you like, without affecting the privacy of the contained message.

The particular mechanism used in the above example is called PGP, aka Pretty Good Privacy, and was created by Phil Zimmerman. (here’s a faq, if you’re interested). It’s what I use, in the form of GPG.

PGP and its relations use a form of encryption known as public key/private key cryptography. The best analogy I can come up with right now is of a castle with two keys to the King’s chambers; one key to lock the door, another to open it. All the security guys can lock the door, but only the King can open it. The “public” key is freely distributable — anyone with a public key can lock the door, and there is no security risk, but only the King has the private key.

In exactly the same way, I have a public key available on www.keyserver.net for each of my main email addresses (put my email into the search box to get the key), which anyone can use to encrypt, or “lock the door to” an email intended for my eyes only. On receiving an email encrypted with my public key, I can then decrypt it with my private key, and read whatever the contents were, or view any attachments.

Currently, PGP encryption is freely available, but still slightly beyond the user-friendly point at which it can be accepted by the masses. I urge you to try it though, if only to give yourself the assurance that your online communication is as hidden as your post. One way to give it a try is to set up an address with www.lokmail.net, which is a free online mail service similar to Hotmail with the added benefit of public/private key encryption. And if you use Outlook, try GnuPG-Plugin, a freely available add-on to Outlook that allows you to encrypt your mails and manage your keys.

Lose ‘em

Back to Governmental Intervention. The media has been brewing up a storm for some time about technology being used to commit crimes. I have written before about the DMCA, which is one example of a very bad law that was passed due to ignorance – certain tools are banned because they could be used to break the law. Imagine banning hammers and chains because gangs use them to fight each other, or banning cars because people can speed in them, or pantyhose because bankrobbers pull them over their heads. I would like you to keep that analogy in mind while considering the following, because a little fear of the unknown is exactly what is allowing the controlling elements of some governments to pass draconian laws without a public outcry.

This USA Today article was released in June of this year, months before Osama bin Laden reached his current level of notoriety. Even the article’s title, “Terror groups hide behind Web encryption”, reeks of a fear of the unknown, this technological monster that aids terrorists. Should the article read “Terror groups use craft knives”, or “Terror groups use demolition explosives” or “Terror groups wield nail files”? These are other tools used by Osama’s crew, but we hear no outcry over their use. Why? Most likely because these are familiar items, things we know and understand in our daily lives, like hammers and chains and cars and pantyhose.

Now here’s the clincher. Encryption is like the envelope you seal your private correspondence in. Well the story about the FBI’s lightbulb looking at your “specially-seethru envelope” is not complete fiction: This article talks of one US Senator and his idea to introduce a “back-door” into encryption software. Another article about the same issue gives an additional insight. Such a back door would be equivalent to the FBI’s lightbulbs spying on your secret correspondence.

Fortunately, I don’t live in the US, nor do most people reading this (I would imagine). US laws don’t stretch as far as Europe yet. An article in the Irish Times in July 2000 discusses a (then new) European law forbidding the decryption by any individual (including the Gardaí, our national police force). This means encryption will keep my communications private indefinitely. Or does it? Much of the technology used on the internet was developed in the United States, and most internet standards are ratified there. If such a law is passed, the encrypted community would suddenly become polarised into those with a backdoor, and those without. I know which camp I would prefer to be in, but knowing the unilateral/isolationist bent of many US leaders, it is likely that such a law would preclude the ability of my software to interact with an american’s software, which means I would be forced to use the special transparent envelopes to communicate with americans.

And hope that the people in control keep a sensible view on how to define a criminal. And don’t move the goalposts[2]. Remember 1984? (the book, not the year). I’d rather not live in a time where Thoughtcrime was punishable. By removing all privacy from my correspondence, we inch closer to that day by day.

Note: this is an article I published in 2001, on an old news site. Leaving it here for posterity. Many links are broken. I may fix this eventually!
Image courtesy of alancleaver_2000, licenced under a CC-A licence

In conversation with a client recently, we discussed the issue of fragmentation in the Java EE space, and how it impacts large organisations. Java EE (previously J2EE) is a set of specifications and standards, with no definitive IDE or application platform. This is by design, and is in contrast with .NET development, which (almost) invariably takes place in Microsoft’s Visual Studio IDE, and deployed to Microsoft’s Internet Information Server application platform runnning on Windows. Java EE systems may, on the other hand, be developed in Oracle’s JDeveloper, IBM’s Rational Application Developer for Websphere, IntelliJ, Eclipse, or NetBeans. Applications may be deployed to Tomcat, BEA Weblogic, IBM Websphere, JBoss, Glassfish, Oracle’s Application Server, and so on.

As an example, consider a large organisation with several development projects on the go at any time, each one started by a different owner/business unit, and potentially hiring individual contractors to get started on the bones of the project, from a spec they’ve created together. The contractor often chooses the framework based on their own experience, and so begins the tie-in for that particular project. As projects mature, their internal development staff take over maintenance of the project, and discover that they have to learn another framework, platform, or even IDE as they transfer between offices.

One solution employed may involve hiring the same large consultancy firm for many of their projects. This ensures consistency — a large consultancy firm will have internal development guidelines and standards — but is quite expensive, so far from ideal. Many organisations hire individuals to provide the expertise required to kickstart a project, and then add their own developers once the maintenance phase arrives. This is where the fun begins.

A modern Java EE application might use JSF or even Visual JSF. Or maybe Struts 2. Or perhaps Spring MVC. Each of these relies on a proper build process, so maybe we’ll use the IDE to build it. Netbeans. Or perhaps Eclipse. Or maybe an IDE-independent build tool, like Ant. Or perhaps Maven. Then we’ll deploy it to a suitable application platform. Glassfish, maybe. Or JBoss. It’s no longer enough just to learn the language; one must also learn the multivarious frameworks, tools, platforms to get one’s work done. One client told me that of the ten developers they had working in their Java EE team, each one had a distinct setup on their workstation. And this was on the same application, all using the same tools and the same platform. So why not just stick with one framework?

Consider Struts. Well, actually, we can’t, and that’s sorta the point. Struts (1) and Struts 2 are significantly different from each other, to the point that migration from Struts to Struts 2 is quite tricky. The developer’s website links to a three-hour tutorial, a three-part guide, and in the Migration Strategies, suggests (as its first option) running Struts 1 and Struts 2 in parallel. So, I think it’s fair to say they are sufficiently different platforms as to be considered separate.

Having said all that, I still think it’s a good thing that this diversity exists. Yup, what I called fragmentation earlier is now diversity. Look at the open source world: the Linux kernel has had thousands of developers working on it, yet is as enterprise-ready (if not more so) than many more monolithic systems. There is no single Linux distribution; Red Hat, Ubuntu, SuSE — all have carved their own very large niches in the industry, and each with a large degree of success. Sure, a project may start and then flounder as another, similar project dominates the niche. Perhaps one or two major players succeed at the expense of others, and we end up with competition in the space, with quality improving on both sides; KDE vs. Gnome, Eclipse vs. NetBeans, .NET vs. Java EE. Evolution is needed to get to that point, and that’s where diversity is a good thing.

Therein lies the challenge for large organisations. The results of such evolution are good for the industry, but the process of evolution is painful. Your modern large organisation is not the instrument of change and improvement; you are not in the business of helping the world decide which framework is the next big thing. You need to make the decision: either reap the benefits of the last generation’s evolution, and back the previous winner, or pick one of the current generation of technologies and stick with it. This is the challenge of conservatism vs. early adoption, but it’s a nettle that must be grasped.

Photo “Diversity Clucks” by chrisjfry

Don’t use anything fancy like Microsoft Project. The trouble with Microsoft Project is that it assumes that you want to spend a lot of time worrying about dependencies….I’ve found that with software, the dependencies are so obvious that it’s just not worth the effort to formally keep track of them.

– Joel Spolsky, Painless Software Schedules

That article has many other great insights, including:

If you are sloppy, and pick big “chunky” tasks (”implement grammar correction”), then you haven’t really thought about what you are going to do. And when you haven’t thought about what you’re going to do, you just can’t know how long it will take.

…and

Many rookie software managers think that they can “motivate” their programmers to work faster by giving them nice, “tight” (unrealistically short) schedules. I think this kind of motivation is brain-dead. When I’m behind schedule, I feel doomed and depressed and unmotivated. When I’m working ahead of schedule, I’m cheerful and productive. The schedule is not the place to play psychological games.

As the article is over ten years old, Joel has end-of-lifed it, and replaced it with this one from 2007. I still like the original!

Photo: “Brian’s Classroom Support calendar” by StanfordEdTech

Another one I couldn’t search up information on, so putting it here for posterity.

I was playing with an elderly blogging platform called Blosxom. It’s really simple to use, although that, combined with its age, means it’s not up to the current status quo. Regardless, I found it fun to play with. Plus, it’s written in Perl, and everything else is PHP these days, so it was a nice novelty.

Now, one of the things it expects (as does the rest of the internet, apparently), is that the CGI file may be treated as an application, folder, or directory on the web. Like this:

http://example.com/blosxom.cgi/2010/06/28#Post

Now, the script itself is the cgi file, a file called blosxom.cgi in the root of my server. As you can see, it looks like a directory in that location, and according to the Internet, that feature should work, passing any “trailing pathname information” (as Apache calls it) to the script via the PATH_INFO environment variable.

Apache didn’t think so.

Now, attempts to google for “apache trailing slashes” or “apache cgi trailing slashes” or anything like that ended up fruitless. It shames me to say I spent over an hour investigating things like ExecCGI, mod-rewrite stuff, and all manner of other things to try to find out why that feature wasn’t working for me.

Eventually, I did what I probably should’ve done an hour before, and looked at the server error logs. Yes, they existed and were visible. I didn’t think that particular host would let me see them — it’s a free host — but I was wrong, and they surprised me yet again.

So, here’s the key: the host had switched off the AcceptPathInfo directive. Ordinarily, that’s left to the handler concerned (CGI, PHP etc.) to decide, which explains the ubiquity of the “on” setting, and the lack of anyone complaining when it’s switched off in various places I looked.

So there you have it, folks. If your CGI scripts don’t work when you have trailing slashes or trailing path info after the script’s name, don’t get lost in RewriteRules – have a look for AcceptPathInfo.

Not only that, but the rest of the audio on the phone stops too, until a reboot of the phone.

Here’s how I fixed it:

• Make a call.
• Note the codec it uses (shown under the call box). It will be something like “PCMA (64kbit)” or “PCMU (64kbit)” or something else technical looking^[1].
• Assuming the call fails (which is, after all, why you’re here!) go into the settings, choose “Audio Codecs”, and disable the codec you just noted. In my case, I set “PCMU (64kbit)” to “Never”
• This is the annoying bit. You’ll probably have to reboot the phone if SIPDroid won’t let you make another call.
• Once you’re back in, make another call, and note the codec. If you’re lucky, you’ll have a successful call, and no lockup. Success!
• …on the other hand, if you’re like me, you’ll have to repeat this process until it works. For me, I hit GSM before it worked.

Of course, there are other known issues at the moment, including one-way audio, issues with registering with the VOIP server, and so on. I didn’t have these issues, so if you are, you’ll have to look elsewhere I’m afraid!

^[1] …where “technical looking” means stuff like G722 HD Voice (64kbit), silk24, silk16, silk8, speex, GSM, BV16, or other supported audio codecs.

(Writing this here because I haven’t seen anything similar after much googling. Hopefully it’ll help someone!)

Other programmers are happy to question the spec, to discover business functionality that may not have been known, and to engage more in design.

Sometimes, and ideally, these functions are separated; you have a programmer who solves problems made clear in the spec, and you have a software project manager/domain expert who designs the solution to be implemented by a programmer.

Having said that, it’s not unusual to find the two functions in a single person, particularly a developer with more experience within the problem domain (in fact, it’s hard to find a project manager who’s experienced enough to design a good solution, who isn’t an experienced programmer).

Standard project management technique would suggest you start with the scoping, and the environment (the “why” and the “what”), and only when you get to identifying the work, to get down to the “how”.

Frequently, the guys involved in organising the scope are not the guys doing the actual nitty-gritty work, so by the time they’re involved, the “why” and “what” should ideally have been specced out.

With software development, particularly with iterative methods, the “what” is usually figured out as part of each iteration, which lets the “why” leak in, and should ideally involve lots of two-way communication between the developers and the client.

This isn’t always how it works though. As I said above, it’s an ideal world that can separate the functions, and often a programmer with enough experience to connect business analysis with programming will already be filling that role in any given project.

You may try this:

…only you find, to your amazement, that SQL Server has put the same number into each row.

Hmm.

In SQL Server, when rand() is called multiple times in the same query (e.g. for multiple rows in an update statement), it usually returns the same number.

Two problems:

* Firstly, the rand() function returns a number between 0 and 1.
* Secondly, when rand() is called multiple times in the same query (e.g. for multiple rows in an update statement), it usually returns the same number (which I suspect your algorithm above is trying to solve, by splitting it into multiple calls)

My favourite way around this problem is to use a function that’s guaranteed to return a unique value each time, like NEWID(), convert it to binary, and use it as the seed.

This works because NEWID() is guaranteed to return a new GUID (a globally unique 16-byte number) each time it’s invoked. We must convert this to binary before using it, as RAND() won’t accept GUIDs as its seed.

So, although RAND() ordinarily gives the same random value for each row in an update, we get over the problem with RAND by giving it a different seed for each row using a function that gives a different result for each row.

Only you’ve got hundreds of employees, each one with their own email address, and your MySQL database is in dire need of updating to reflect this.

To get around this, you’ll need to replace the relevant part of each email string within an update statement, grabbing the hostname substring (after the ‘@’) with a REPLACE, and replacing it.

Note: REPLACE() is case-sensitive, so if needs be, you can use LOWER(email) inside the REPLACE function if you need to catch all case possibilities, as below:

This will also convert all your email addresses to lowercase, so be aware of that.

Accessibility is a hot topic on the web, and there are many emerging standards to help move in an accessible direction. We have CSS media types, alternative web pages with simpler navigation and high-contrast styling, alt and title tags, and the general move away from mixing style with substance.

Along with all of this, we have the relatively old standard of using the accesskey attribute on invokeable elements such as hyperlinks and form inputs. This allows us to attach a keyboard shortcut to elements in our webpage.

Unfortunately, this isn’t a widely used, or easily implemented standard. Although the accesskey attribute is widely supported, only one commonly-used browser (Opera) at the time of writing provides an easy way for users to see what accesskeys are enabled on a given site, and there is no widely-accepted standard for choosing which accesskeys perform which function.

However, with a little jiggery-pokery we can implement a simple way to show accesskeys on demand:

The code for the above is pretty simple. First, the button:

<input type="button" value="Show access keys" 
       onclick="showKeys()" style="float: right;" />

The hyperlink itself (abbreviated):

<a href="http://jeremysmyth.com/...." 
       title="Link to this post" 
       accesskey="9">This link</a> 
will bring you...

Finally, the javascript:

function showKeys() {
        if (document.styleSheets) {
                var sheet = document.styleSheets[0];
                var len = sheet.cssRules.length;
                // reformatted to fit
                sheet.insertRule("[accesskey]:after {" +
                        "font-weight: 700; " +
                        "border-bottom: 1px blue dotted; " +
                        "content: '[' attr(accesskey) ']';}" , len);
        } 
}

Clicking on the button calls the “showKeys()” function in the Javascript script block, which adds a style to the current stylesheet. The style automatically styles elements with the “accesskey” attribute, adding the value of that attribute after the element itself.

Put simply, it adds a styled [9] after the hyperlink, because (1) it has the “accesskey” attribute, and secondly, the “9″ is the value of that attribute, as calculated by the attr() function.

Note: The above Javascript won’t currently work in Internet Explorer; for that, you’d need Microsoft’s addRule function rather than the standards-compliant insertRule I’ve used.

We have thousands of open idea requests (the codebase is nearing 20 years old), and close only a sizeable fraction of those opened regularly.

From our perspective, the idea requests are welcome, but not all of them are actionable; some are brilliant, and are implemented immediately because they work well with our vision; some are entirely incompatible and are closed/denied.

The majority fit in between; they’re ideas that would work with a bit of tweaking, or a bit of thought, but aren’t necessarily on the primary development roadmap, so don’t get our attention immediately. Nor do they warrant closing, because they are relevant, merely not timely or important.

Because our developers have their own ideas, their own neverending todo lists, we treat the open idea pile more as inspiration than as a roadmap. There’s very much a feeling of “we’ll get to it when we’ve run out of other things to do”, but this never happens in practice.

I know it’s a cop-out not to choose one or the other, but I think it’s a normal thing to have to choose between two equally bad things in a public forum like this: either responding to most feature requests with a “denied”, and so risk upsetting the folk who love the community enough to contribute with their own ideas; or leave some of them dangling because they’re not immediately and obviously wrong, but to do something worthwhile with them takes more time and effort than the idea deserves right now.