Today's idea is another one which is simple to state, but which (due to the lack of data interchange standards) is far from simple. What I want is a personal database that keeps tracks of everything on my computer, and everything I ever see on the web, in newsgroups or in email.
This came up because I recently had to find an address for a party I was invited to. The trouble was, I couldn't remember how I had gotten the invite, so I didn't know where to look for the address. If it was a telephone invite, I would have made note in a file, or on my online Yahoo calendar, depending on how rushed I was at the time. Had it come by email it would be in an email folder for all mail from that sender. Of course, there was a half dozen folks that I know that might have invited me, so it could be in any of those folders. It was also possible I had seen it on one of the various online forums where its possible to send a private note to a group of people. I never did find the address, and had to phone someone and have them look it up in their notes.
Clearly, there has to be a better way of organizing things. What is needed is a free-form database that will sit in the background and watch all email, newsgroups, new files and everything else I add to my system, and keep notes. When I burn a CD it should note what files I've put on it, and what its serial number is. It needs to do that all without ever prompting me or asking any questions. So, the next time I need to find an obscure article about a new ink-jet system I saw six weeks ago on the net, I can do a search on my personal database. Since it will have indexed every word of every textual file on my machine (.pdf, .doc, .txt etc) and every word I've seen on the net, I should be able to quickly find what I'm looking for.
Of course, in order to do this it needs to be able to monitor all sorts of things I do on my machine. It needs to index files as I save them, and it has to be able to read my email and see what I'm browsing on the web. It will also need hooks into my CD burning software and other storage solutions that I use. Finally, it has to be able to run with minimal user setup and intervention. I must not need to know what a database schema is, never mind being asked to customize one. It should do its job silently by default, although it should be possible to augment its data in a simple way.
For example, when I'm reading my email, I should be able to click on a missive and add a note or set of attributes to it, so that I can find it more easily later. Similarly, when I burn a CD, it should be possible to call up a database window in which I can associate a CD title with the serial number, to aid in later finding it on my shelf.
Ideally, of course, the database would entirely replace the file system on my computer, but that is a more massive undertaking than I am proposing. What I am really asking for is something like a beefed-up Microsoft File Indexing system for XP, except that I want one that works. To this end there are some features it must have:
- It needs to be open source so that folks can extend its API in useful ways as folks want to make it able to gather and index new types of information which weren't planned for in the original design. This will also facilitate it becoming a standard.
- It needs a way to plug-in modules for dealing with various open-source and proprietary data types. That way, for example, Wordperfect could be encouraged to produce a module that reads their file types, without making their file format public.
- It needs to be invisible when not needed and easy to use when searching for something. It must have a reasonable UI that caters to both the novice and the power user, without condescending to either — in other words, no cartoon avatars.
- You must be able to search for individual words in titles and contents, attributes and other various metadata, and be able to do complex boolean searches.
- It must be simple to backup and restore, and able to deal with such activities as migrating a partition to a larger hard drive.
- It needs to have a good data export/import system and a data interchange API. There are too many personal databases out there that handle only one type of data, and which cannot talk to any other databases. This makes them almost useless.
- Although its not a priority, it would be nice if it didn't use up outrageous amounts of disk space. Disks are getting cheap, but whatever amount it does use up, it should more than pay for the resources it uses in added convenience.
As I said before, it would be better (from various standpoints) to have this integrated into a file system so that noticing when a file is renamed or moved is more straightforward, but database file systems have been promised for both Linux and Windows for years now, and neither appears any more closer to existing than when I first heard about the idea. Maybe this interim solution will help spur development in the right direction.