Divegeek's Stuff

Routines

2011-08-22T10:52:00.000-07:00

I've never been very good at routines. Not because I object to doing the same thing every day -- if I like what I'm doing, I don't mind doing a lot of it -- but because I'm too easily distracted and not disciplined enough to maintain an orderly schedule.

However, I've come to believe that this is a failing, and one that limits my ability to accomplish as much as I want to. There are always too many different things pulling on me and my time, and without routines I tend to spend lots of my time on things that don't matter at the expense of things that do. Also, after all of the chaos of the last couple of years, I find myself actually looking for routine.

Routines are especially important, I think, for making time for those things that we know we should do, and some of the time want to do, but which in the moment are often easy-to-skip chores. Exercise is a big one. Scripture study. Flossing. Keeping up with the finances/budget. And lots more things, some big, some very small.

So, I'm going to try to create a nice set of routines for myself and stick to them, at least for weekdays. I'll work on weekends later. So here's what my day should look like (I don't expect anyone to read this; typing it all out is for my benefit):

5:00 AM. Wake up, get the kids up, then catch up on e-mail while Kris gets breakfast ready.
5:30 AM. Breakfast & scripture study with the family. I won't eat, though.
6:00 AM. Hop on the bike and ride 10 miles to the bus stop in Longmont.
6:38 AM. Get on the bus, putting my bike on the rack on the front.
7:14 AM. Arrive at the office, shower and change (and brush, floss, comb hair, etc.). Also weigh in.
7:45 AM. Spend an hour focused on improving my work efficiency. This means reading books, learning tools, etc. Whatever I can do to make myself more effective. A couple of days per week I'll spend this time editing photos instead. From time to time I may opt to do the full 23-mile bike ride to the office, rather than taking the bus, which will consume some of this time.
8:45 AM. Breakfast in the Google cafe.
9:00 AM. Work.
12:00 PM. Lunch in the Google cafe. If the weather is nice, eat outside and get some sun, or take a walk.
12:45 PM. Work.
5:01 PM. Hop on the bus to Longmont.
5:41 PM. Arrive in Longmont, start the 10-mile ride home.
6:15 PM. Arrive home.
6:30 PM. Dinner with the family.
7:00 PM. Flexible. In the near term it'll mostly be organizing the house, getting the computers set up, etc. I'm sure there will be plenty of helping the kids with homework, and such. On Fridays, date night with Kristanne.
8:30 PM. Chat with Kris, check-in with Makae, plan the coming day.
9:00 PM. Kris and the kids go to bed. Update accounting, check e-mail, personal scripture study.
10:00 PM. Bedtime.

That's my plan. It includes a little more than eight hours per day of work, which is realistic. Google doesn't demand the hours that a lot of software companies do, but my experience is that I need a little more than 40 hours per week to keep up on everything. The decision to bike/bus rather than drive means I'll spend three hours per day commuting, rather than the one hour I could spend, but about half of that time will be exercising and the other half I can use to read, do e-mail, or whatever. Riding 20 miles per day should mean that I really don't need to worry about going to the gym or other exercise (though I probably should do some pushups, situps and maybe lift weights a little -- gotta set up the weight bench in the basement).

Wish me luck.

Web site password manager

2011-06-25T22:55:00.000-07:00

If you're like most people, you use a bunch of different web sites, including some that handle some pretty important stuff -- like money -- but you use the same password for all of them. And it's probably not a very strong password, either.

That's a really bad idea. If one of those sites gets hacked, the attacker could get your password and then start trying other places it might work. Essentially, the security of all of your web accounts is only as strong as the weakest one.

But the reason we do that is because it's just danged hard to remember a whole bunch of different passwords. I'm hopeful that eventually OpenID will solve this problem but in the meantime I found another nice solution: Passpack.com.

It allows you to store all of your passwords in one place, and it even provides a little button you drag onto your bookmarks that automagically types the right password for each web site you visit. So, I've now gotten rid of my one password for everything and replaced it with a bunch of passwords, one for each web site I use -- and I never have to actually type any of them. I just go to, for example, "discovercard.com", then click the "Passpack it" bookmark and I'm automatically logged in. And the password is something like 3yPtdkogzh8H, which I could never remember, but don't have to. Or even something like "ò-èuÓ¿¸>8ÝAÖ" for sites that can accept the strange characters -- I don't have to type it so there's no reason to limit it to something I can type.

This *also* works from any computer that has a web browser. I do have to remember my "passpack key", which is what secures all the rest of my passwords, but as long as I know that, I can log into passpack from anywhere and then use it to log into other web sites. Do be cautious about using passpack from public computers, though... there are ways that a deeply hacked computer could extract your passwords if you unlocked them while using it.

Finally, I also had Kristanne set up a passpack.com account, and then I "shared" my passwords with her. So she can use them to log into web sites, too, so that we both have access to our bank account, credit card accounts, etc. If I ever change the password for one of them, I just update it in passpack and then when she uses it to log in she'll get the right one -- and won't even know that it changed because she never sees or types the passwords.

How all of this works is pretty complex, and very technical. Suffice it to say that it appears to be done right from a security perspective. Your passwords and your passpack key are never sent to passpack.com in "plaintext". Software running in your browser applies a salted secure cryptographic hash function to your passpack key to produce a 128-bit AES key which is then used to encrypt your passwords. The encrypted passwords are sent to passpack.com and stored there (which is how it can work from a browser on any computer -- they encrypted copies are downloaded). Password sharing is done by generating a 1024-bit RSA key pair which is used to exchange AES session keys to make it possible for people you share your passwords with to use them. Actually, you can also use the sharing service as a form of highly-secure e-mail.

If the preceding paragraph is gobbledygook, just trust me. Or if you'd like to understand it, just ask sometime and I'll explain it in as much detail as you could possibly want :-)

If that paragraph isn't gobbledygook to you, and you see a bunch of possible holes, that's because I left out all of the details.

Won't people freak out and call the police if I openly carry a gun?

2010-12-20T06:51:00.001-08:00

Short answer: It's possible, but not common.

Utah is a very gun-friendly state, and many residents of Utah have some familiarity with guns. In addition to that, people tend to assume that the calm, confident actions of a normal-looking person must be legal and most people are, frankly, oblivious to most of what goes on around them.

These factors combine to mean that open carriers find that the vast majority of people simply don't notice the gun, especially if it's not large or ostentatious, the majority of people who do notice the gun don't react to it and most of those who react ask questions or make comments rather than calling the police. People who open carry regularly report that the most common interactions are very positive ones, where the carrier has an opportunity to explain the issues to the curious or just engage in “gun talk”, chatting about shooting, firearms and lawful self-defense.

That said, negative interactions do ocasionally occur. Most often these take the form of store or restaurant employees asking the open carrier to leave. Sometimes these requests are provoked by questions from other customers but most often they are initiated by employees. The appropriate response of the open carrier to such a request is not to argue or debate the point, but simply to comply, ideally without completing a purchase and with a followup letter or e-mail to explain the carrier's point of view and to request clarification of the store policy.

In rare cases the police are called and become involved. In some areas, 911 operators are being trained to question “person with a gun” reports, asking what the person is doing, if he or she is threatening anyone, etc. In those areas, police are unlikely to be dispatched. If police do arrive, interactions are generally peaceful, though not all police officers are fully informed on the law so some may make false statements regarding the legality of open carry. If you are the recipient of police attention due to open carry, please stay calm, stay polite and if the officer is making incorrect statements, just say “I'm sorry officer, but I don't think that's correct. Can you please ask your supervisor to verify it?”.

If you are a person who has an unusual style of dress, hairstyle or appearance, especially if your appearance is one that might be associated with gang or criminal activity by many people, or if you appear or act intimidating or frightening even without a firearm, you are much more likely to have negative interactions and police interactions as a result of open carrying. This may not be fair or right, but it is reality. Also, young people and those who engage in body modification (piercings, tattoos) are more likely to have negative experiences with open carry. In contrast, well-dressed, well-groomed, responsible-appearing people tend to have no issues.

I don't have a concealed carry permit, can I carry a gun?

2010-12-20T05:34:00.001-08:00

It is legal to carry a firearm in Utah without a permit, but there are some things to watch out for if you leave your own property or vehicle. Specifically, if you're carrying in public your firearm must be unloaded and unconcealed, and you must avoid school zones and buses or trains.

It might seem that carrying an unloaded gun is useless, but Utah's law has a peculiar definition of the word “loaded”. Specifically, the law says that gun is loaded if either:

There is a round in firing position; or
A single mechanical action will cause the gun to fire.

This means that you can carry a semi-automatic with a full magazine as long as the chamber is empty, because an empty chamber means there is no round in firing position, and it takes two mechanical actions to fire the gun: rack the slide and pull the trigger. For a revolver, the chamber lined up with the barrel must be empty, and if the revolver is double-action, meaning a pull of the trigger will rotate the next chamber into position, cock the hammer and fire the gun, then the next chamber must be empty as well.

In contrast to this odd definition of “loaded”, Utah's definition of “concealed” is very straightforward. A gun, or any other dangerous weapon, is concealed if it is hidden so that people aren't aware of it but it is readily accessible. In practice, the best way to open carry is to use a normal outside-the-waistband belt holster. I recommend using a holster that has some “retention” features, which are devices that lock the gun into place so it can't easily be removed by someone who isn't familiar with the holster, and I recommend practicing techniques to stop a gun grab attempt. Such attempts are extremely rare, but your gun is your responsibility so it makes sense to be prepared.

The final issue to watch out for is locations where it's illegal for you to carry. The most frequently-encountered issue is school zones, but carry is also banned on UTA buses and trains and in bus and train stations. There are other areas that are off-limits even for permit holders, including courthouses, jails, secure areas of airports, etc.

Without a permit, it is illegal for you to carry within 1000 feet of any school property, where “school” means any pre-school, day-care, elementary, secondary, or post-secondary institution. Since day care facilities are often ordinary homes with no distinguishing marks, and post-secondary schools like cosmetology schools and massage therapy schools are found in many strip malls and may not look like schools at all, this means there are a lot of places that carry without a permit is illegal. The area within 1000 feet of a school-sponsored activity is also considered a school zone.

If it weren't for some exceptions in the law, school zones would make lawful unlicensed carry almost impossible, and would pose serious problems for owners of homes and businesses near schools. First, it is legal to possess a firearm on private property located within a school zone, with the owner's permission. Second, it is legal to possess a firearm in a vehicle located within a school zone, with the driver's permission -- as long as federal law enforcement isn't involved. The federal school zone law allows firearms in vehicles, but only if the gun is completely unloaded and in a locked container. This law has never been enforced since it was passed in 1995, but it is on the books.

Relevant laws:

UCA 76-10-505 bans carrying a loaded weapon on a public street. Note that the law does not define “public street”, so to be safe I recommend interpreting it as “anywhere but private property with the permission of the owner”.
UCA 76-10-502 defines when a weapon is deemed loaded.
UCA 76-10-504 bans carrying a concealed dangerous weapon.
UCA 76-10-501 defines “dangerous weapon” in paragraph (6), “concealed” in paragraph (3) and “readily accessible for immediate use” in paragraph (15).
UCA 76-10-505.5 is the state law that bans carrying a dangerous weapon (including firearms, per 76-10-501(6)) in a school zone.
18 USC 922(q) is the federal law that bans carrying a firearm in a school zone.
UCA 76-10-1504 and UCA 76-10-1507 ban carrying a firearm on a bus or in a bus terminal.
UCA 76-10-1503 defines “bus” to include any transit vehicle, including trains like TRAX and FrontRunner.

There's going to be an increase of activity here

2010-12-16T08:36:00.000-08:00

I need to get my utahlegalcarry.com site up and running soon, but in the meantime I've decided to start posting here some of the content that I ultimately want to put there. Eventually I want to build a fairly comprehensive FAQ list for people who want to understand legal carry of firearms and other dangerous weapons in Utah. To begin with, I'm going to start posting essays on various topics here, with the intent that they'll eventually get incorporated into my FAQ list, and so that in the meantime I can easily refer people to them.

I'm going to shoot for adding one or two articles per day here, so expect to see some activity.

Here's a partial list of issues/questions I'm going to address:

Politics/opinion (note: I'll give my opinions, but try to present all sides)

Shouldn't only police officers have guns?
Isn't it a bad idea to allow anyone who wants to carry a deadly weapon?
Doesn't allowing people to carry guns at our schools endanger the children?
Doesn't owning, possessing or carrying a gun make you more likely to get shot?
Why would anyone want to carry a gun?
Why would anyone want to carry a gun openly, if they can conceal it?

Legalities of carrying a firearm or dangerous weapon

I don't have a concealed carry permit, can I carry a gun?
I have a concealed carry permit, where can I carry?
How come Utah courtrooms don't provide firearm storage lockers, like the law says they should?
I have a concealed carry permit, does my gun have to be hidden?
Can my city or county ban dangerous weapons?
I'm a student or staff member at a Utah university, can I carry there?
My employer has a policy against carrying weapons at work. Do I have to obey that policy? What might happen if I don't?
What kinds of weapons does a Utah Concealed Firearm Permit allow me to carry? How many can I carry?
What should I do if I see a "No Firearms" sign somewhere?
Can I carry in other states? If so, which ones and what are the rules?
Can I take my gun on a plane? A train? A bus?

Practical issues around daily carry

What kind of gun should I get?

What caliber is best?
Are revolvers better than semi-automatics?
What brand of gun is the best?
Are big guns better than small guns?

Is it important to use a holster?
What kind of holster should I use?
Should I use a lot of different holsters?
What should I do with my gun when I go to the bathroom?
If I'm pulled over by the police and I have a gun, do I have to tell the officer, and if so how should I do it?

Firearm safety

What's the difference between an accidental discharge and a negligent discharge?
How can I make sure that my guns don't hurt anyone?
How can I make sure that my guns aren't a danger to my children?
Are there any special safety precautions I should take while carrying?
Is it safe to carry a gun with a round in the chamber?

Securing firearms

Am I responsible for keeping my guns secure?
How can I secure them in my home?
How can I secure them in my car?
How can I secure them when I travel on airplanes or trains?

Training

How much training do I really need? Isn't the Concealed Firearm Permit course enough?
What kinds of training are available for free?
What kinds of training are available for fee?

Legality of self-defense

Who am I allowed to defend?
When is it legal to shoot someone?

On the street
At home
In my car
On my property

Do I have to try to escape before I can defend myself?
When should I shoot someone?
When should I NOT shoot someone?
Should I threaten someone with my gun? Should I draw if I'm not planning to shoot?
Is it necessary to carry a non-lethal defensive tool, like pepper spray?
If I get in some sort of altercation that doesn't involve my gun, should I call the police?
If I get in some sort of altercation that does involve my gun, but I don't shoot anyone, should I call the police?
If I shoot someone, what should I do next?

What should I do before I call 911?
What should I say to the 911 operator?
What should I do while waiting for the police to arrive?
What should I do when the police arrive?
What should I say to the police?
Will I be arrested? If I am, what should I do?

Can I be sued even if the police decide I didn't commit a crime?

Utah's Concealed Firearm Permit

Who is eligible to get a permit?
What records does the state examine to determine eligibility?
What's the process for getting a permit?

How much does it cost?
How long does it take?
What are the steps?
Can I get a temporary permit quickly if I'm in danger?

How long is the permit good for?
What is the process for renewing a permit?
What kinds of things might cause my permit to be suspended or revoked?

I'm sure plenty more stuff will come up as I flesh this out. I'm going to attempt to use the newspaper article structure for each answer, meaning that the article provides the basics first then incrementally expands the details, so the reader can stop at any point.

Awesome new voting system used in Maryland

2009-11-05T10:44:00.001-08:00

I'm something of an election geek. I deeply believe that although democracy sucks (including the slightly-less-sucky Republican variant we use), it sucks less than any other system of government. But to make any Democratic system work there must be a good way to determine the will of the people, and that turns out to be very hard to do.

There are two main parts to the hardness. First supposing you can collect everyone's opinions, how do you put them together to make decisions? We generally use the "plurality rules" system; whichever option gets the largest number of votes wins. That system is full of problems, but that's not the subject of this post.

The other problem is collecting everyone's opinions and doing it accurately. Someone who can manipulate the elections has tremendous power, and there are a LOT of ways of manipulating elections. Stalin famously (and apocryphally) said "Those who cast the votes decide nothing; those who count the votes decide everything," and historically that is the BEST way to manipulate an election. Let the people vote as they will and then either just report whatever result you want or, if you're more subtle, just tweak the counts enough to give the desired outcome.

In the US we've recently gone through a somewhat mind-boggling electoral change by deploying electronic voting machines. I say mind-boggling, because these machines are the PERFECT tool to facilitate large scale, undetectable election fraud, and supposedly we're a country that takes its elections very seriously.

If the idea that electronic voting machines are a bad idea strikes you as surprising, or the view of a Luddite, let me point out that I'm a professional software engineer with a career focus on building extremely high security systems. If you wanted to build a very secure electronic voting system, I'm the kind of person you'd hire to do it -- and I would tell you that it is a TERRIBLE idea. And not just me. A few years ago, as the debate was just starting to heat up, several hundred of the world's top computer security experts collaborated on and signed a paper which basically said the same thing. They explained in detail why computer security technology at present and for the foreseeable future was simply not up to the task. The very best in the world said, in effect, "Not only don't we know how to make such a system secure, we really doubt that it's even possible."

By the way, the system that Utah uses is the worst of the worst.

This debate, however, did provoke a bunch of top-flight cryptographers, mathematicians and computer scientists to start thinking hard about how to design an election system that is secure against fraud. They wanted to build something that cannot be manipulated by the people in power without the populace being able to detect it, and which also preserved the anonymity of voters. It's very easy to build a secure voting scheme if you don't mind revealing who everyone voted for, but that enables vote-buying and coercion.

Some very cool ideas were developed, notably originating from Ron Rivest (who is the "R" in "RSA", for those who know what RSA is) and David Chaum (the creator of some very cool untraceable digital cash schemes). It's probably unfair to mention only their names, because there were many contributors, but they were the main "well-known" researchers involved. The first versions of these ideas were theoretically very cool, but completely impractical. Lots and lots of academic security ideas start out exactly that way.

A few years ago, some of the disparate ideas started to come together, along with some refinements that made them more practical. The result was a system called "Punchscan", which was used to carry out some real-world elections for university student body leadership. It was a good test environment because the stakes were relatively low, but the scale was big enough to make the test realistic. A system that works for a few hundred people is a lot different from one that works for tens of thousands, and once you get to that level it's really not that different to scale it up to millions.

The third-generation successor of Punchscan, called Scantegrity II, was used on Tuesday to carry out the municipal elections in Takoma Park, Maryland. This marks the first time these new ideas in election integrity and verification have been applied to a real government election. And it worked well. The security experts said that electronic voting machines were a bad idea, and now their recommended alternative was used for the first time.

The basic goal of the system is to ensure that it is impossible for ballots to be lost or modified undetectably, and to do it in the context of a practical election that is cost-effective, easy to run and easy for voters to use.

It uses optical-scan ballots, with a small twist. The "bubbles" on the ballot are pre-filled with invisible ink. The voter has a marker that contains the chemical that activates the ink, so swiping the marker across the bubble causes it to be visibly filled -- mostly. When the ink is activated, not only is the bubble-filling ink revealed, but a three-letter code is also revealed. The code is "printed" by the absence of the ink.

A voter who wishes to make sure that his ballot is counted correctly takes a moment while in the voting booth and copies down that three-letter code, as well as the ballot serial number. After election results have been posted, he then goes to the election web site, types in his ballot serial number (which isn't associated anywhere with his identity), and make sure that the codes he copied down are among a small list of codes presented to him. If they are, then he knows his ballot was submitted and tallied as part of the final count.

Now, HOW that code's presence on the web site verifies that the ballot was counted correctly is a little bit complicated, and I'm not even going to try to explain it here. If you want to know, a good starting point is this paper. Also, that code-verification is far from the only thing that has to be done to verify the integrity of the election. Ballots have to be audited before and during the election (audited ballots are not used to cast actual votes), there's a pre-election "code commitment" process that the election officials use, and a post-election "randomized partial checking" system that verifies that the ballots submitted and tallied were actually tallied correctly.

The system allows all of these various integrity verification processes to be carried out by ANY interested parties. It's presumed that the candidates and political parties will take part in them, and that democracy watchdog groups will as well, but any interested person can do it. In fact, the system is designed so that anyone who wants to can easily conduct a personal recount of the entire election, just by downloading the data over the Internet and then running some software on it. The software has to do some complex things, including a lot of fancy math, but the detailed specification for what the software must do is available, along with mathematical proofs demonstrating how it assures integrity, so that anyone with the relevant skills can write their own software to do the verification. Those who don't have the skills to write the software just need to get a program from an organization they trust, and use that.

All of this makes it possible to prove to an arbitrary mathematical probability that the election results were accurate. By "arbitrary mathematical probability" I mean that there is always a possibility that the election was fraudulent and that the fraud was undetected, but for any given margin of error you can decide how much auditing needs to be done to make the probability of fraud as small as you want it to be. The smaller you want to make it, the more auditing is required, but you can achieve any desired level of certainty.

In addition, the Scantegrity system allows for manual recounts of the paper ballots. Manual recounts are actually less reliable than the statistical verification the system uses, but they're an option, unlike with electronic voting systems (including Utah's; no one has ever managed to successfuly recount an election from those paper rolls the machines produce).

This is very cool stuff. Of course it doesn't address issues with registration fairness, fraud, etc., or issues with whether or not plurality voting makes sense, and certainly does nothing to address voter apathy, lying politicians, media influence or any of the many other problems with the political process, but it DOES allow us to have confidence that our votes are counted as cast, which is something that has never been very certain, and which the current generation of voting machines has made very, very questionable.

Software Copyrights

2009-10-07T13:28:00.000-07:00

I've written about this various places before, but I thought I'd put it here, mostly so I can link to it from other places rather than retyping the arguments.

I'm a big fan of copyright law. In the abstract, at least -- there are a lot of problems with our current law. And computer software is a huge part of my life. It's not only my job, it's one of my major passions and lately my biggest hobby. So it's natural that I should be interested in how copyright applies to software, and I think that the way we use copyright for software is very, very broken.

To explain why, first I have to give a little background on copyright.

The idea of modern copyright is a pretty simple one: Society grants creators of all sorts of useful and artistic intellectual works control for a limited period of time over who is allowed to produce copies of their work. There's a little more to it, and lots of corner cases and caveats, but that's the basic idea. Other than the "limited period" part of it, pretty much everybody understands that if an author writes a book, you can't make copies of it and sell them on the street without permission.

But, why not? Why do we do this? Thomas Jefferson argued that copyright made no sense. He said that ideas are naturally infinite, that as they're passed from person to person, everyone is enriched. He compared it to candle flame and pointed out that a man who lights his candle from mine has obtained light, and I have lost nothing.

So why should society invest large amounts of money and time in enforcing copyright laws, which restrict the natural freedom and urge to share?

There's a really good reason, and it's the one described in Article I Section 8 of the US Constitution: "To promote the progress of science and useful arts, by securing for limited times to authors and inventors the exclusive right to their respective writings and discoveries". Copyright law was created as a means to convince creators of intellectual works to publish them, to make them available to the world. The way it works is that society removes the freedom of everyone but the creator to make copies, for a time, so that the creator can benefit from his or her work. But any benefit to the creator of the work is just a pleasant side effect, because the real goal is to get that work published, into the hands of as many people as possible where it can spark new ideas and inspire new creativity. In other words, where it can Promote Progress.

Oh, it also promotes progress by motivating people to write, sing, etc. But the real goal of copyright was to promote publication and dissemination, because that's where the real progress is made, when ideas build on other ideas.

So, how does this relate to software?

Well, copyright law is the primary legal tool used to control the distribution of software, both by individuals and corporations who are trying to make money, and even by the Free Software movement, who have other goals. But Congress never really sat down and thought hard about how copyright should apply to software. Congress did change the law in order to balance out the short-term and long-term advantages to society when other technological changes came about, but they didn't do it fast enough for software, so the courts ended up deciding for them.

The courts basically decided that software is sufficiently expressive and creative to qualify for copyright protection. I completely agree. I have seen some truly beautiful code in my life, and even written a little. I hope someday to take a photo that's as beautiful as the best of the code I've written.

But what the courts failed to do, in a crucial oversight, is that they failed to distinguish between "source code" and "compiled binaries".

Source code is what programmers write. Reading or writing it requires some training, but it's designed to be human-readable, and computers can't make any direct use of it. Instead, the source code must first be processed by another program called a "compiler" (which itself was written in source code and processed by a compiler). The compiler turns the source code into "machine code", the actual pile of computer instructions that the computer reads and follows, like a very sophisticated cake recipe.

Source code is also what programmers read. Just as authors of poetry and novels hone their craft by reading the writing of other authors, the best way for a programmer to learn new ideas and new techniques is to read the code written by other programmers. The progress of software is promoted by making sure that programmers can read what other programmers have written, not so they can copy their programs directly, just as authors don't make word-for-word copies of the works of other authors, but so they can pick up ideas. Structure, phrasing, word choice, dialogue, character development plot arcs... all of these are things that authors learn from other authors, and there are analogous concepts to all of them in software.

The difference between a book and a software program, though, thanks to those first court rulings that decided that copyright applies to software, is that it's impossible for an author to publish a book to the world and simultaneously keep secret from the world the words that he used to write it. In order for an author to reap the benefits of publication, he also has to allow other authors to read his words and learn from them. How could it be otherwise.

With software, it is otherwise. Programmers can read and learn from source code, but once that source code has been fed through the grinding maw of the compiler, turning it into an opaque mass of machine instructions, it is extremely difficult to examine the result and determine how it does what it does. Not impossible, but very difficult. It's perhaps akin to taking a book and running it through a paper shredder, then piecing it laboriously back together in order to read it.

But copyright law, as currently applied, protects that ground-up version just as much as the readable version. And with software, the ground-up version is the one that has value to non-programmers. So, individuals and companies can produce software, publish the opaque binaries on CDs, in boxes on the shelf in the local computer store, and never have to reveal the ideas they used to create it. And yet they get the full weight of the legal system standing behind their copyrights, even though they have sidestepped the whole purpose of copyright law, to promote progress by disseminating ideas.

Moreover, although copyright is supposed to last for a limited time, after which copyrighted material falls into the public domain and becomes available for anyone to use for any purpose they wish, the source code of software published in binary-only form will never see the light of day. The binaries will fall into the public domain, but the source code was never published and will be lost.

It's not a total loss, of course. Other programmers can often infer interesting things about the structure of software from the behavior of binary copies. And some 'reverse engineering' (figuring out how it works by poking through the binary) does take place. But progress is hugely slowed by the predominance of 'closed' software, software for which no source code is available.

The Free Software movement is really a reaction to that limitation on progress. And it's a significant testament to the progress that is enabled by openness that Free Software constructed by ad-hoc groups of volunteers around the world often not only compares with, but bests, similar "closed" software constructed by large, well-paid and focused corporate teams.

I think the solution to this problem is very simple, though politically challenging: Software makers should be required to publish source code in order to receive copyright protection. It would still be illegal for programmers to copy this copyrighted source code, and illegal to copy the ground-up binaries as well, but other programmers could read the code and learn from the ideas, satisfying the progress-promoting goal of copyright law.

There would be practical benefits as well. As a purchaser of a software package, you would have some assurances that you do not now have. For example, should the company that sold you the package collapse, you could still hire a programmer to fix any defects you find in the software (the legalities of that would have to be worked out, but at least the ability is present). Even before you buy it, you would probably have the ability to ask others who've purchased it what they thought of the program -- and not only it's outward behavior, but also its inward structure. A good programmer can tell a lot about the quality and reliability of a software package by examining its source code. This is similar to a mechanic taking a look under the hood to see if an automobile is sound. But with closed-source software, the hood is welded shut.

A non-obvious benefit that I'm convinced we would see is a reduction in the amount of code copied illegally between programs. How can that be? Doesn't only publishing code in binary form completely prevent illegal copying? Not really. There are still people who see the source code, and they can still copy it. Lots of programmers (illegally) take a copy of their work with them when they change employers. I personally have witnessed a couple of cases of stolen code incorporated into closed-source software (not while at IBM; IBM is exceptionally cautious about this).

So illegally copying of source code happens now, but how would making source code more widely available reduce it? Simple, because it would be easier to find. Right now, companies that illegally copy source code usually get away with it, because they distribute only opaque binaries and it's difficult for anyone to recognize their act. But if everyone published source code, finding illegally-copied code would in most cases be a simple matter of scanning. There are tools right now that can scan a body of source code to see if it contains any code taken from the thousands and thousands of open source programs in the world. Companies use these tools to verify that their programmers haven't lifted some open source and dropped it into the company's software as a time-saver. It's easy to see that if most commercial source code were available, that this approach would be easily extended to cover that as well.

There might still be some companies who have such important and novel ideas in their source code that they dare not publish it. They would also have an option to protect their assets, through another facet of intellectual property law: Trade Secrets. They could classify their source as a trade secret, and sell their software only to customers who are willing to sign a contract committing them not to make copies. The contract would protect the binaries from being freely redistributed, and the source code would stay a secret. Any employee or other person who knowingly divulged the secret source code would be guilty of a crime.

I'm convinced that if obtaining copyright protection for software required publishing the source code, we would see an explosion in software progress. The quality and capability of the software packages we all use every day would grow by leaps and bounds. Software technology researchers would have access to a huge body of code to analyze and learn from, to help cull the best techniques and processes to help all programmers be able to do a better job. Tools would improve. Quality and reliability would improve. Security would improve. There would still be lots of problems, of course, because software is inherently, fundamentally hard. But applying copyright law in accordance with its underlying principles would better serve society's interests as a whole.

GridBackup Re-architecture

2009-10-04T07:48:00.000-07:00

GridBackup still isn't fully functional, but I think I'm going to change directions already. So far, it does scanning and backups pretty well, and has a basic backup verifier so you can check that your files really are safely backed up to the grid. It doesn't have a restore tool yet, but I could slap a simple one together in a few hours if it were needed.

However, there are some problems with the way it works now.

First, it doesn't address laptop users well at all. I have set it up for my brother Dirk, but because his backup server is just a machine sitting in the corner, there's really nothing for it to back up. All of his important files are on laptops. I set up a folder on the server, accessible via Samba (Windows file sharing) so that he can drop important files in a place where they can get backed up, but, predictably, having an extra step like that means that backups don't get done.

Second, there are problems with the implementation. The sqlite database used by the GridBackup, GridUpload and GridVerify scripts doesn't handle concurrent access well, so you can generally only run one of them at a time. But you have to be a little careful (in ways that I only know from experience, and am not sure I could explain) or you can corrupt the sqlite database when stopping one program to start another. The uploader really needs to be a daemon ('service' in Windows terminology), just running in the background all of the time. Ideally, it should really be integrated into Tahoe, so it starts and stops with Tahoe. I don't think I want to do it in Tahoe, at least just yet, because I want more freedom to work. However, given the Twisted application plugin system, I may be able to write it as a plugin that can be added to Tahoe.

In the new architecture, the "uploader" becomes the "backup server". It's purpose is to accept backup jobs delivered to it by a backup client (i.e. the "scanner"). The client should be run wherever the files are (i.e. on your laptop), and would deliver any changed files to the server as fast as it can. The intention is for the server to store those files itself until it can get them safely uploaded into the grid. From the client's point of view, once it delivers the files to the backup server, they are backed up, though it may take some time for the files to actually be delivered to grid storage.

One issue this approach raises is that it requires the backup server to have room to temporarily store all of the files sent to it. This shouldn't be a huge problem in most cases, because the assumption is the backup server is the same machine as the Tahoe node, and it has to have a lot of storage available -- usually 2-3 times as much as what the clients that use that server wish to back up.

However, in some cases users may not be using their Tahoe node for storage, and so may not have a lot of storage available. In that case, they'll want to run the backup client and server on one computer, and configure the server to restrict the amount of storage it uses, and to attempt to get files from their "original" locations, rather than its own storage when it doesn't have enough to hold everything. There's still value in having it store recently-modified files, on the grounds that if they've changed recently, they'll probably change again soon and the backup server doesn't store its own copies they may change before it gets around to uploading them to the grid.

Orson Scott Card's "Empire"

2009-06-28T06:53:00.000-07:00

I just finished reading (actually, listening to) a 2006 novel by Orson Scott Card called "Empire". It was a good story, as his always are, and thought-provoking, as his usually are.

I highly recommend reading it, so I won't spoil it by giving away too much of the story here, but it's set in the present, and is a story about potential civil war in the United States, more or less between the radical left and the radical right (exactly who starts it and why is something of a mystery through most of the book, so I won't spoil it by being more specific). Card also includes an after word where he discusses his reasons for writing the book, and that's what I wanted to post about.

Card points out the inanity of the liberal/conservative dichotomy in our current political structure. There are many issues facing the nation, and the conservative and liberal ideologies are bundles of ideas about each, bundles that don't really have any kind of internal consistency or reason. For some reason you can't be in favor of the Iraq war and believe that homosexuals should be allowed to marry, or pro-gun and in favor of amnesty for illegal immigrants, or... you get the idea. Abortion, gay rights, foreign relations, drug policy, fiscal policy, taxation, gun rights, environment, welfare, intellectual property... there are a lot of different issues, and many of them are completely unrelated. Yet if you espouse, say, the "conservative" position on one issue, you're automatically assumed to buy into the rest of the conservative bundle. And if you really want to fit in, politically, you'd better not stray too far from one party line or the other.

Even worse, as Card points out, is the vitriol that the two major parties direct against one another and against anyone who doesn't agree with every point in their position bundle. From my perspective Card overstates the problem, but I understand why. He's a fairly public figure who writes a lot of political opinion pieces, but his beliefs don't clearly fit into one camp or the other, so he gets railed at and even despised and ridiculed by both.

I see this all the time on slashdot. Whenever the topic of Orson Scott Card comes up, and it comes up often given that he's one of the great science fiction authors and slashdot is loaded with scifi-loving nerds, there are always dozens of posts slamming and ridiculing him for opposing gay marriage and calling him a conservative extremist. These posts, just like the pundits we see on TV, boil his statements on the topic down to single-sentence summaries and even one-word labels, neither of which do any justice to the nuances of his real opinions. They're so oversimplified as to be simply wrong.

Meanwhile, if his name comes up on the pro-gun forums I also frequent, he's equally reviled and despised there as a hated liberal, because he supports gun control and is critical of free-market capitalism.

The real crux of his complaint, though, isn't so much the bundling of issues but the level of fear, anger and distrust that we direct against anyone who chooses a different bundle than we do. And the thrust of the book is that those extreme emotions are so powerful in the radical wings of the two parties that it's not inconceivable that it could lead one side or the other, or both, to decide that they must take up arms against the other. He addresses the question of whether or not it's possible for a civil war to occur when the sides are so intermixed, without clear geographical boundaries, by pointing out the examples of Rwanda and Yugoslavia, where exactly that happened.

I don't know if I fully agree with his conclusion about the feasibility of a right/left civil war in the United States. In his examples the sides were divided by religion and/or ethnicity which are more powerful than political ideology. But after reading the book I have to grant that it's possible. I know for a fact that radicals on both sides of the divide do believe that those on the other are actually evil, and it's not inconceivable that some of those extremists might take up arms in sufficient numbers and with sufficient organization to create a war.

And once the shooting starts, if it's big enough, then all the rest of the people may well be forced to choose sides, because if they don't then both sides will consider them the enemy.

War aside, though, I also agree with Card that the approach that both of the parties are taking is destructive. They're so divisive and so vitriolic that real discussion of the issues is nearly impossible. The media doesn't help any, either, since their idea of a "balanced" discussion is two partisans angrily talking over one another.

From my perspective, the whole thing would be amusing if it weren't so sad. I simply don't care about most of the issues they get so up in arms about, and where I do care my positions don't align at all well with either of the bundles. What I do care about, deeply, is the centralization of government power and the importance of the rule of law and the Republican style of government. But neither side really talks about that. And given the current system there is simply no place for voices that don't align with one side or the other. If you're not a Democrat or a Republican your opinions may as well not exist for all of the public discussion they'll get.

Lotsa wasted space

2009-06-17T10:51:00.000-07:00

While experimenting with some of my GridBackup code last night, I noticed something very interesting: There's a lot more wasted space on my hard drive than I thought.

Oh, I knew there were tons of files I haven't touched in years, and may never look at again. What I hadn't realized was how much stuff is duplicated.

GridBackup has progressed to the point where the file system scanning and backup log generation is pretty solid (though I still need to test on Windows). I'm working on actually uploading data to the grid now.

Anyway, I decided to put my queue of jobs to be uploaded into a database, for convenient access and lookups. One of the things stored for each job is a hash of the file. For those who don't know what that is, it's a kind of unique fingerprint. Different files basically never have the same hash, and identical files always have the same hash. So, by counting unique hashes in the database, I can tell how many unique files are on my computer.

It turns out that out of the 733,189 files on the computer, only 520,242 of them are unique. So over 200,000 files are duplicates of at least one other.

Of course there are a boatload of empty files, over 11,000 of them. There are many reasons that programs create empty files, so that didn't surprise me. But there were a lot of others, too -- all told I have over 1 GB of space wasted due to duplicates.

I generated a list of the culprits, sorted by total space wasted (size * (count-1)), and while some of worst offenders were my own fault, lots of others had been quietly generated by various programs, wasting a lot of space.

What to do? Nothing, of course. This is 1GB wasted out of nearly 200 GB of stuff, on a 300 GB drive. But I thought it was interesting, and it reaffirms my suspicion that the fact that GridBackup tracks files by content, rather than name, will mean that lots of stuff will only have to get backed up once, even though it may exist many times in the group of computers.

Distributed vs Centralized Version Control

2009-05-28T23:11:00.000-07:00

I've been meaning to write about this for a while, because I get asked about it on a regular basis, and because I wanted to try writing it all out to help me clarify for myself exactly why I find distributed version control so much better. The time to write it and the motivation to do so finally coincided, so here it is.

The basic differences between distributed and centralized version control are:

1. DVCS doesn't necessarily include any notion of a central repository. Every developer has a full repository with all version history, etc. on his local machine. Often, teams choose to designate one repository as the "main" one, just for convenience, but it isn't necessary.

2. Sets of changes can be pulled easily and flexibly between repositories, and there are powerful tools for picking out particular change sets.

Those are the core differences, though git (and most other DVCSs) also handle version control at a whole-tree level, rather than per-file. Due to the way change sets are pushed and pulled between repositories, it becomes important that all of the file changes related to a logical change (bugfix, feature, etc.) are a single commit that is thereafter managed as a unit.

The different approach to version control results in different interactions between developers, just as locking-based VCSs are used differently from merge-based VCSs. With a centralized repository, every developer pulls updates from the repository and pushes changes to it. That same approach can be used with a DVCS, but often is not, especially with OSS projects.

A DVCS also lends itself to a "request-pull" model. A developer fixes a bug or implements a feature and commits it to his reposiory, then notifies other developers (usually the project leader or his delegate) that the change is available to be "pulled". That person can then pull the changes, vet them for appropriateness, style, functionality, etc. and if he likes them, commits the changes to his repository.

Very large projects (hundreds of developers) normally end up organized in a tree structure, with a central "official" repository managed by a project leader plus a second layer of "lieutenants" (typically responsible for major subsystems), who often have one or more layers of trusted assistants themselves. New code is developed by a "leaf" developer in his local repository and when it's sufficiently clean/stable/etc., he notifies an appropriate individual who decides whether or not to accept the changes. That individual aggregates changes from many people, and then notifies the next layer up in the tree, and so on.

The most visible project of that sort is, of course, the Linux kernel, which has over 500 regular contributors and thousands of occasional contributors. With the DVCS model, Linus Torvalds is able to "manage" this huge development team, and to successfully vet, test, and integrate megabytes of changes ever month.

Smaller projects adopt different models depending on the level of team cohesiveness and trust. Within corporate teams, cohesiveness and trust are high, and the DVCS is used in a centralized mode, with every developer pushing changes into the primary repository without going through another individual. Open source projects are often a hybrid, with a small number of core developers who have direct access to the "official" repository, and a larger number who don't, and must ask one of those core developers to pick up their changes.

Increasingly, though, even small open source projects are abandoning that mode and shifting to almost entirely decentralized operation. "Almost" because there still ends up being an official repository which is the one from which releases are pulled to be made available to non-developers. Often, though, the owner of that repository is purely a "release manager", who may not even be a programmer.

Github really facilitates this very loose model. If you want to modify a project hosted on Github, you do it by "forking" the project. Github tracks those forks, though, and when you commit changes, those changes are made visible to the project owner, who has the ability to pull them. It also provides a one-click mechanism for requesting that the owner pull a change set, as well as very nice tools for handling the merges.

In my opinion, DVCSs offer two main advantages over centralized VCSs:

First, the organizational flexibility. Whatever sort of structure makes sense for your project and your team, you can implement it, simply by deciding who pulls from whom and, less commonly, who is allowed to push into whose repository without oversight.

The second is lower-level, more pragmatic: It's just really, really convenient to be working out of your own, purely local repository. Want to work on an airplane? No problem, you have full version history, the ability to commit, roll back, branch, merge, etc. Most all DVCSs provide you the ability to consolidate, divide and reorder commits to "clean up" the version history, as well. That may sound like a bad thing, but it's not: It allows the developer to present change sets as logical, cohesive, coherent wholes, in spite of the fact that the true development process involved many false starts and backtrackings. That not only makes moving change sets between repositories easier and cleaner, it makes the version history MORE USEFUL.

Finally, there's one more aspect to this practical benefit, which accrues most prominently to git: Speed. Linus Torvalds (primary author of git) likes to point out that beyond a certain point performance isn't just a pleasant reduction in thumb-twiddling time, once an operation is so fast that it takes no human-perceptible time, it changes the way developers work.

Perhaps the best example is branching. Many CVCSs provide branching and merging tools, but they're often cumbersome to use, and slow. Creating a new branch on a large project may take a minute -- or sometimes much more. Switching between branches is similar.

With git, for example, branch creation is instantaneous, and shifting between branches, even on trees with tens of thousands of source files, takes well under a second. As a result, git developers tend to use branches for everything they do. Any new little project, bugfix, feature request, etc., spawns a new branch. If the change works out, it's trivially pulled into the main line (and pushed/pulled from there to other developers). If not, deleting a branch is as trivial as creating one (However, NO commit is actually lost so it can always be brought back if really needed).

With traditional CVCSs (and even some of the DVCSs), developers end up with many copies of their source tree, in an effort to keep different streams of work separated. One copy may be for a new feature that's under development on the current in-development version. Another might be a copy of that tree, used for a risky, experimental approach to the new feature. Another might contain the released version of the software, used for testing. Yet another might contain an older version, with changes to fix a bug. With git (and a good build system that doesn't rebuild code unecessarily), there's no need for more than one.

So, to summarize, DVCSs are better because they can do everything CVCSs can do, and more, and faster. If you want to use a DVCS as a better CVCS -- or even as a front end to a real CVCS; git-svn makes Subversion hugely better -- you can do that. If you need a different model, you can do that. And you can do it all faster and, once you learn how, more eaily.

Pathname handling, coninued

2009-05-27T21:14:00.000-07:00

So, here's the solution:

On non-Windows NT systems, I read and manage all pathnames as raw bytes, basically ignoring whatever encoding the system thinks it's using. Since the common case is to restore files to the same sort of system they were backed up from, that ensures that every file name can be backed up and restored.

But what about the less common case, where the restore is to a different sort of system, with a different encoding? To address that, in addition to the raw bytes of the pathname, I also store the file system pathname encoding -- that tells me the way the system claims to interpret the raw bytes of its names. When doing a restore, the restore process checks to see if the target system uses the same encoding as the source system. If so, it just writes out the raw bytes. If not, then it decodes the names with the source system decoder and encodes them with the destination system encoder.

If there is a "bad" name (a name that is invalid per the source system's encoding rules), that decoding may fail, or it may just produce garbage. If it fails, the restore system logs the error and retries the decoding with a special mode that replaces invalid characters with an "unknown" symbol, so that the decoding and restore succeeds, even though the file name is damaged.

However, the original byte string is still in the backup data, so theoretically I could someday make tools that allow the user to specify the encoding, or that try a bunch of different encodings, or whatever. The data exists to recover the correct pathname, assuming that's even possible.

There's one more issue, though: The method I'm using to write file names into the backup log only takes valid Unicode (and encodes it with UTF-8). So I want to store and manage raw byte strings, but I need to convert them to Unicode. To do that, I "decode" the byte strings with the latin1 decoder. That's a widely used encoding standard that has the convenient property that it maps every possible byte value to a unique Unicode value. The results are probably complete garbage, but that's okay, because I only use this latin1 encoding as a transport method. During restore, I decode the UTF-8 names to get Unicode, then encode them with latin1 which gives me back the original byte string.

GridBackup pathname handling

2009-05-25T22:14:00.000-07:00

(BTW, this post also serves as an announcement that I'm resuming work on GridBackup, barring any more crises)

After struggling with various approaches for months, I've finally settled on a method for handling file and directory names. You would think this would be easy, but it's not.

In a nutshell, representing names is a hard problem, and one that's been bothering computer scientists and software engineers for decades. To a computer, everything is ultimately a number, so finding a way to store the word "hello" requires converting the text to a series of numbers.

No problem, we just assign a number to each letter (h = 104, e = 101, etc.) and store that list of numbers, right? Well, sure, but the problem is that there are a LOT of symbols that may be in names. All of the first major computer systems were invented in the US, and so they had no ability to handle any symbols other than the ones we use -- basically A-Z (upper and lowercase), 0-9 and assorted punctuation characters.

When Europeans got involved, they needed to expand this set a bit. They needed various accented characters, plus a few modifications like the French ç (note the little tail on the bottom). Okay, so define numbers for each of those, right?

Oh, but then we have the cyrillic alphabet. And Indian Devanagari, Gujarati, etc. And Arabic scrip. And Korean Hangul script. And then there are the various Chinese and Japanese writing systems, some of which don't even have alphabets, but are ideographic.

The net is that there are a lot of writing systems in the world, and computers ultimately need to support all of them that are used by any significant number of people. And not only do they have different sets of symbols, but they flow in different directions (left to right, right to left, top to bottom) and many have various complex ways of merging or joining characters.

There is now a system for defining numbers for all of those many, many symbols, and specifying how they connect together, etc. It's called Unicode. Now, if only the whole world used it (and all used the same variant of it -- there are two major versions, UCS-16 and UTF-8, plus some more minor ones), then we could all pass files around and everyone would see the names correctly, whether they could make sense of them or not.

Actually, there is one major segment of the computing world that does use Unicode religiously (specifically UCS-16): Microsoft Windows NT, which includes Windows 2000, XP and Vista. This is one thing that MS got right, mainly because NT came about in the mid 90s when all this stuff was pretty well understood. On Windows, every file name is guaranteed to be correct, valid Unicode.

Mac and Unix systems (Mac lately is Unix, though not traditionally) came earlier and used different systems, as did other versions of Windows before NT. Specifically, they used a wide variety of encoding systems developed in different parts of the world, each appropriate for the writing of that area. In order to make the software customizable for different areas of the world, they provided a way to set a "locale" or a "codepage" which basically said "On this computer, we represent file names using CP949", or similar (CP949 is a numbering of Korean characters).

That works, but it means if you lift a file from a Vietnamese computer and drop it on an American computer, the name is complete gibberish. I don't mean gibberish in the sense of "not understandable if you don't know Vietnamese", I mean "not understandable, period", because the numbers used to represent the Vietnamese characters would be interpreted by the Amerian computer as whatever those numbers mean in the American character set. In many cases, the numbers representing a name in one system may be invalid on another computer. The computer won't just display garbage, it'll give you an error.

Even worse, many of these systems allowed users to specify their own "locale", so one computer may have one user using a German character set while another uses Russian.

So, the upshot of this is that on Mac, Linux, Unix and older Windows systems, there may be files with names that make absolutely no sense when interpreted the way everything else on that computer is interpreted.

The question is: How should a backup program deal with this?

I ultimately settled on these key goals:

The backup should ALWAYS succeed, regardless of whether or not the file names are "valid".
The backup should preserve enough information that it's just as possible to figure out the correct characters for a name from the backup as it is from the original system.

In addition, I have a constraint: The format that I'm putting the file name information into in the backup system (JSON) only accepts Unicode. So I have to somehow make sure that everything I store is valid Unicode, even though I don't necessarily have any idea what characters are in the name.

It's late. Having explained the problem, I'll describe my solution tomorrow.

PeerBackup status

2009-02-20T07:36:00.000-08:00

The file system scanner is pretty much complete. I'm satisfied that it doesn't miss anything and that it runs fast enough. The initial run on a file system is slow, of course, because it has to hash every file on the computer. On my desktop machine, it plows through 182 GiB in just over two hours. After that initial run, though, it's pretty quick. It does a complete scan in about five minutes; a little more if there are some big files that have changed. Not too bad. I imagine there is still some performance tuning I can do to squeeze a little bit out of that time (I notice that a little-has-changed run maxes my CPU, so there's probably some inefficiency there), but I'll defer that until after I have the basic system working.

On the upload job management, I have a good start. I've implemented a workable system for prioritizing uploads, and come up with a potential default prioritization scheme that balances three prioritizations -- user vs. system files, file age and file size.

Some files are more important to back up than others. Ideally, what we really want is a fairly fine-grained mechanism for allowing the specification of classes of files and then each class should be completed before the next is begun. So, for example, I want my personal finance information backed up before anything else, followed by my photos, followed by my work projects, followed by my personal projects, followed by system configuration information (/etc and some stuff in /var) followed by everything else in my home directory, followed by locally-installed applications (/opt and /usr/local), followed by everything else.

I ultimately want to allow such fine-grained control, but I don't expect very many of my target user audience will understand enough about their computers to set it up. I'm also not sure I know enough about Windows or OS X to define good approaches for those.

So, for now, I'm starting simple. My algorithm strongly prefers files in /home, /Users or C:\Documents and Settings, whichever of those paths exists, and doesn't prioritize beyond that. This essentially creates three classes: user files, other files, and files that shouldn't be backed up at all (implemented by specifying exclusions on the scanning process).

The next prioritization element is modification time. It seems like a good idea to back up recently-modified files before old files, on the theory that they're of greater interest to the user. The function was selected so that files modified in the last minute get maximum time prioritization, files that are a week old get 50% and it trails off from there, getting down to 10% after a year.

The last element is size. If you have to pick between getting a whole bunch of small files backed up or a few large ones, it's probably better to get the many small files. The function I chose is designed to be at maximum for empty files (though those won't actually be backed up, obviously), at 50% for 1 MiB and trail off from there. Oh, and the size value used is the lesser of the file size and the size of any cached delta for the file, so small deltas will get high priority. I'm not sure if that's a good idea or not, since it will tend to favor adding more revisions to backed-up files over getting files that haven't been backed up yet into the grid.

All three elements are weighted equally, on a scale from 0 to 1 million. But the "user files first" element is boolean -- either the file gets the full million or else it gets nothing -- while the age and size factors will almost never give full value for a file, and tend to trail off very quickly. I've only done rudimentary testing, but it looks like only very young and very small system files end up prioritized over user files. It'll need tuning, but I think it's a good start.

Job queue scanning is implemented, but I have realized another piece is required. Although I can create a priority queue containing all the jobs (and it's reasonably fast, even on a big queue), I need to add some logic to detect multiple jobs referring to the same path, because some of them may have dependencies on others.

If a file has multiple jobs that are each deltas from a previous revision (which means that one full revision is in the grid already), it doesn't make sense to upload them out of order, because later deltas are useless without their predecessors. Because the scanner won't cache deltas for files that haven't already been successfully uploaded, the question about uploading a delta whose full revision basis hasn't been uploaded should never arise.

So, I need to add a mechanism to allow me to identify when an upload job is one of several referring to a file, and then decide how to handle them.

I'm hoping (again) to get an alpha out this weekend, but I think I'll be snowboarding with the kids on Saturday, so we'll see how it goes.

PirateBay

2009-02-19T08:50:00.000-08:00

The PirateBay trial is generating a lot of discussion all over the world. I've posted my thoughts in bits and pieces on a couple of different forums, but I want to pull them together into one place.

First of all, I want to make clear that I support copyright. I think it's a fundamentally good idea when applied for its original purpose (original purpose in the US, at least, the original original purpose was censorship by the British Crown). That purpose is expressed in Article I, section 1 of the US Constitution this way: Congress is given authority to pass laws to "promote the progress of science and useful arts, by securing for limited times to authors and inventors the exclusive right to their respective writings and discoveries".

Note that the purpose stated is to "promote progress", not to compensate authors and inventors (and musicians and filmmakers and programmers and...). The Framers recognized no inherent right of people to get paid for creating stuff, and certainly no right to control and profit forever from a single piece of work.

By nature, there is no natural limit to the flow of ideas and expressions, and therefore no inherent sort of "ownership". As Thomas Jefferson put it "He who receives an idea from me receives instructions himself without lessening mine, as he who lights his taper at mine receives light without darkening me." The natural state of "intellectual property" is that it doesn't exist, and there's no obvious reason it should, because my obtaining a copy of something in no way takes anything from anyone else.

However, the Framers also recognized that much of value to society might lie forever hidden because there was no motivation to publish it. Publishing books was expensive, and if someone else could copy them there might be no way to recoup those costs. In the 18th century publication created a significant barrier to the flow of material into the public domain, and so Congress was authorized to do some things to motivate people to cross that hurdle. They didn't concern themselves so much with things that might never be created for lack of motivation, but that's an issue as well. A talented author who must do other work to feed himself won't be able to write as much, and that's a bad thing.

Enter the notions of copyrights and patents (and trade secrets and trademarks as well, though they're a little different). Copyrights and patents work very differently, but the basic purpose of both is the same: To encourage publication, so that ideas and expressions can enter our culture, and to do it by granting artificially-created and enforced limited monopolies.

It's important to remember the artificial nature of these monopolies, and the fact that society spends significant amounts of money on enforcing them. The reason that we do this is not to benefit creators (though it does, and that's a nice side effect), but to benefit society, to enable the production and distribution of material to the public, and to ensure that the material will one day (soon, hopefully), enter the public domain and become a basis for the creation of even more.

With that understanding of copyright, it should be clear that as publication gets cheaper and easier, there less and less reason for society to provide motivation to people to publish. In fact, arguably, the publication barrier has disappeared entirely. I can write this little treatise on copyright law and publish it to the world at absolutely no cost other than my own time to type it. I could make a movie and publish it on youtube, or via bittorrent if youtube isn't sufficiently high-quality.

Technology has also lowered the cost of production of many kinds of copyrighted materials, but that tends more to raise the bar in terms of expected quality than to lower the cost. Talented people work hard to make things, and society benefits if there's a way they can get paid to put their talents to work.

Okay, with that as background, here's my take on the PirateBay situation:

First, from a technical perspective, there is no way to argue that the operators of the PirateBay engaged in any sort of copyright infringement. They basically provide a specialized version of Google, one that provides links to information hosted lots of other places around the world. Further, they don't even provide the links, they just operate a service where users can post the links and other users can find them.

This means that any sort of action against them is over actions that are a couple of steps removed from the real lawbreaking. "Contributory infringement", "conspiracy to make available ifringing copies", and the like are the phrases we have to use. I just don't think it holds water. If they're convicted, then we also have to force Google to verify that none of its links point to infringing material -- which is really hard given that whether or not a certain piece of audio, video or text is even infringement is sometimes a subject of debate even among lawyers.

In general, although I think copyright is a good idea, I don't think these sorts of generations-removed charges are a good idea, just as I don't think it's a good idea to hold gun manufacturers liable for what bad people choose to do with the tools they make. Tools have good and bad uses, and the PirateBay hosts a lot of links to perfectly legitimate content as well as links to stuff that is copyright infringement in most jurisdictions.

The bottom line is that if the motion picture studios and record labels want to shut down infringement, they need to go after the infringers, not the people who made the tools the infringers chose to use.

But there's a deeper issue here, and this is why I went into the origin of copyright first.

The deeper question is whether or not it should be illegal to share movies, music and books. It's generally been understood for years (and even upheld in court) that sharing stuff with your friends is not actionable copyright infringement. In general, in the past the line between infringement that mattered and infringement that didn't was whether or not anyone was making money. Want to tape a song off the radio? Fine. It's infringement, but non-commercial and non-impacting. Now make 10,000 copies of that tape and start selling it on the street, and you'll have police and lawyers knocking on your door.

In the past, that worked fine. The big media companies may have grumbled behind closed doors, but the non-commercial sharing was both hidden and moderately small-volume.

Now, however, non-commercial sharing has exploded. The Internet makes it possible for me to share a CD I bought with tens of thousands of people around the world. To media execs, who mentally count every copy shared as a sale lost, that adds up to billions in lost revenue. It's the kind of thing about which Something Must Be Done.

As a shareholder in some of those companies, I agree. As a member of society, I wonder. The purpose of copyright was to motivate people to overcome the publishing barrier, but that barrier is gone. At this point, the only remaining purpose of copyright is to help ensure that stuff gets created. Banning filesharing is probably good for Viacom's bottom line, but does it really help to motivate artists, musicians, authors, etc.?

I don't think so. The different industries have different dynamics, so lets look at each one in turn.

Book publishing first, just because its the industry that seems to contain some of the most forward-thinking and englightened people. I suppose it shouldn't be surprising that authors tend to be a little more deep-thinking that musicians or moviemakers. And I suppose it shouldn't be surprising that Science Fiction authors and their publishers are a little more forward-thinking than others in their industry, which leads me to Baen. A few years ago, Jim Baen decided to try an experiment. He offered the authors he publishes to option of putting a few of their books on-line in a FREE libray. Anyone could download them, and everyone was encouraged to share them with whoever they wanted. There were no restrictions on copying, and they were provided in every electronic format Baen could think of, so they could be read on any computer, electronic book reader, cellphone... whatever. Or you could even print a copy and read it on paper.

Everyone in the industry scoffed, of course, and some were angry because they thought Baen's decision to give books away for free would hurt their sales as well as Baen's. Jim Baen, and his authors -- some of the current top-selling Sci-Fi authors -- stood firm, though and carried out their experiment.

It's actually incorrect to call Baen's Free Library an "experiment", though, because Baen didn't see it that way at all. He saw it as way to prove a point and -- even more importantly -- a way to make some money. He fully expected that he and his authors would profit handsomely from giving stuff away for free.

Eric Flint, one of the authors in question, likes to point out that the book he decided to put up, Mother of Demons, was his first novel, and far from his best, and both its quality and his obscurity showed abundantly in its poor sales. It was a profitable book, but not what you'd call a success. Not until it became the first book on the Free Library, anyway. It is now Baen's best-selling backlist title, and one of Flint's top few.

That result surprised even Baen and Flint. They expected that giving away some of an author's work for free would increase the sales of other works. They calculated that their expected profit would come primarily from giving the first book in a series away, and then selling the rest. What even their bright, inquisitive science-fiction author minds never expected was that the books they chose to give away would also see significant new sales. But it was consistently true. So much so that Baen has since made a habit of including CDs in their hardcover editions, which contain completely unencumbered copies of dozens of books, with a label on the front that says "please share".

Why does this work? I have a number of theories, but they all boil down to this, at bottom: People like to buy stuff that they like. So if you're selling entertainment, the main problem you have is helping people to find out that they like your stuff, and giving things away for free is a good way to do it.

Turning to the music industry, the first thing we should take note of is that the industry has been giving their product away for free for decades. And not just "giving", either, they've actually been paying people to give their product away for free. They want so much to pay people to give their product away that we've had Congressional investigations and criminal prosecutions specifically to stop all this paying, and yet the industry has continued to do it, becoming ever more creative in the ways they pay people to give away their product for free.

I'm talking about radio, obviously. The record labels long ago understood that the very best way to increase sales was to increase airplay. Why? Because people rarely buy what they haven't heard, and never buy what they haven't heard of.

For some reason they think file sharing is different. They don't see a tune played on the radio as a lost sale, but they do see a downloaded MP3 as a lost sale. Are they right? Is it different? Not according to every formal, published study that's been done. There have been plenty of them conducted around the world and they invariably find a few things:

The people that download the most buy the most.
Increased on-line sharing is correlated with increased sales.
There is no concrete evidence that filesharing decreases sales.

This all seems to make no sense, until you remember that people like to buy stuff they like. How many music fans don't want to own the actual albums of their favorite band, with cover art and inserts and all? My kids have complained about me giving them music purchased on allofmp3.com for birthdays and Christmas. Why? The music is the same, what does it matter? Because it's NOT the same as owning the "real" CD. And the fact is that publicity is what music needs for success, and filesharing provides that publicity.

Now, there is some question as to whether or not this will continue to be true, as we migrate more and more to digital file-based players (iPods) as the primary way we listen to music. Is there a difference between buying a song on iTunes or downloading it from a filesharing system? Perhaps not, but I think there is. The difference is that people who have the money like to buy stuff they like, and they feel good about having done it.

Another finding of several of the filesharing studies has been that individuals' entertainment budget is a relatively fixed quantity, and that they'll spend all of it. At most, the ability to get some things for free may shift where they spend it, but they'll still spend it. This indicates that a sale "lost" to filesharing (meaning a case where someone downloads a song and doesn't buy a copy also) isn't really lost . Maybe it's shifted. Most likely it wouldn't have happened at all anyway.

But let's suppose than in a future world where no one listens to music on anything but iPods, and no one buys it in any way except downloads, and all downloading has migrated to free sites, will music disappear? Will musicians have no way to make a living.

Consider Jonathan Coulton. He's a talented singer and songwriter from New York who writes and performs offbeat, fairly geeky music. Stuff like a song about zombies titled Re: Brains, with the chorus "All I wanna do is eat your brains, we're at an impasse here, maybe we can compromise", or a song about a love-struck computer programmer called Code Monkey. Coulton writes, sings, performs and records all of his music himself. He gives many of his songs away for free through his web site, and sells others for $1 each. He not only makes no bones about fans sharing his music (even the songs he doesn't give away for free), but he licenses all of it under a Creative Commons license which allows other people to use his stuff for free to make other stuff, like those videos I linked to.

Coulton has no record label deal, and doesn't want one. He quit his job as a computer programmer and spent six months writing and recording music, publishing a new song for free in every edition of a weekly podcast. By the end of a year, he was not only making a living doing his music, but a significantly better living than when he was working a normal job before -- and all of this in spite of the fact that it's trivial to get all of his music without paying a penny.

He doesn't have gold toilet seats, but he does have a comfortable income well into six figures, doing what he wants to do, and this in spite of the fact that his music is rather niche, with a very narrow market appeal. Personally, I think that's a fantastic model for music in the future, and a friend of mine pointed out that by allowing musicians to earn relatively "normal" incomes, rather than raking in millions, we may cut down on the number of them that crash their Ferrarris or fry their brains with drugs, and actually get more good music out of them.

But, in case you may be thinking that this approach only works for niche acts, consider Radiohead. They recently performed an experiment with their new CD, allowing listeners to download it and then pay whatever they think it's worth. I downloaded it and hated it, so I didn't give them a penny. They haven't released exact figures, but apparently the experiment was a huge success. Regular CD sales of the album were the highest Radiohead has ever experienced, the album going platinum in its first week on the shelves, and word is that their on-line sales were just as good, in spite of the fact that it was basically an on-line "tip jar", with no obligation to pay anything. According to the band, most on-line buyers paid pretty close to the normal retail price.

So, while obscure Jonathan Coulton makes a fine white collar-class living by not bothering himself about copyright, Radiohead is making millions doing the same thing.

With both books and music, it's quite clear that aggressive copyright enforcement, lawsuits against file sharers, lobbying for draconian criminal penalties for infringment, etc., are all both bad for society and bad for authors and musicians.

But what about movies? While one guy in a home studio can write, perform, record and distributed music, making a movie costs millions of dollars. Often hundreds of millions of dollars. Yeah, Blair Witch only cost a few tens of thousands, but that's not the rule, and not the quality of movie we usually want.

Were file-based movie watching to become the dominant form, that might be a real problem. Luckily, at present I don't see any indication that movie theaters are going away. Again, it comes down to people like going out to the movies, and that experience can't be replicated by file sharing. File sharing could concievably cut into DVD sales, but even if it wiped them out (which is highly unlikely), all that means is that moviemakers have to focus on the box office as their primary revenue source. That's what they did for decades until the advent of VCRs, and there's no reason the model can't work again. In actual fact, I expect that cinephiles, like audiophiles and bibliophiles, will continue buying because they like to. Especially since the moviemakers often include additional benefits, such as nice boxed sets, posters, extra content on the media, etc.

So, given that all of these industries seem well-positioned to prosper in the presence of file sharing, what value is there to society in investing a lot in limiting its members' freedom to share when they want to? Note that the focus is on costs and benefits to society, NOT costs and benefits to those who create the media. Their well-being is a part of the calculation, but only insofar as the fact that if they can't make a living making the stuff we want, they'll have to spend their time doing something else, which we probably consider less valuable.

Bringing this back to PirateBay, I think that they should get off because they didn't do anything wrong under the law. But not only did they not do anything illegal, they didn't even do anything wrong. Copyright is an artificial construct which we prop up through legal means in order to achieve an end goal. Even if the PirateBay facilitates infringement of copyright, our copyright law itself is currently badly broken, because it hasn't incorporated the new Internet reality. A copyright law adapted to achieve the maximum societal benefit at minimum societal cost will allow non-commercial file sharing.

First we need to stop bugging these people for doing what they're doing, and then we need to fix our broken laws. Unfortunately, the large media companies have a tremendous amount of influence over our laws, and filesharing costs them money. In fact, as far as the music industry goes, there's good reason to believe that the Internet makes the record labels as necessary as buggy whip makers. More and more musicians are connecting directly to their fan bases and not bothering with a label at all, so the people who are the focal points of the industy are fighting a desperate rearguard action for their own survival, precisely because they're not essential.

We need to help our legislators understand these issues, or the media industry is going to drive laws that will fill our prisons with college students whose only crime was to do something that harmed no one.

Unit testing vs defensive coding

2009-02-14T13:13:00.001-08:00

While working towards releasing the first alpha of my PeerBackup application, I decided to check out the code coverage on my unit tests. In the process of addressing some deficiencies, I noticed an unexpected tension between defensive coding and unit testing.

I want to get my test coverage as close to 100% as possible, because I really like the confidence it gives me that the code is working correctly. This is particularly important since I'm using Python, and without any sort of static validation, if a line doesn't get executed I have absolutely no idea whether or not it is at all functional. It could reference nonexistent variables or functions; there may even be some syntax errors that won't be found until the line is actually run.

So 100% coverage is a Good Thing. But... I learned years ago that defensive coding is also a Good Thing. I often write a few lines of code to address cases that I'm pretty sure are impossible, and which I definitely can't think of a way to create. Since I can't think of any way to create the case, I can't write a unit test to cover it -- which means that my unit test can't be 100% unless I remove the defensive code.

As a result, I find myself tempted to remove defensive code that is probably a really good idea, because there just might, after all, be a way to trigger it. Or even if there isn't, perhaps some future change will create a way to trigger it.

On balance, I'll leave the code in there and accept less than perfect coverage. I don't like it, though.