The Meta Demo

The Meta Project is so large that many man-years will be required for anything like a full-scale implementation. On the other hand, if the basic ideas are sound, it should be possible to implement a subset which demonstrates their essential validity.

So, I have begun development on the Meta Demo (www.cfcl.com/Meta/md_help.html). The demo is a specialized "search engine" for information on operating system software. Given a manual page name, file system path name, or keyword specification, the demo retrieves a set of related items (e.g., path names and man page names), complete with terse annotations.

At present, the demo covers only some base distributions of FreeBSD and Mac OS X. Coverage of the Ports Collection and/or other operating systems are under active consideration. I am also looking into the issue of providing browsing capability into the files themselves.

Despite its severely limited charter, the demo has proven to be both interesting and useful. Even as an experienced Unix user, I find that the demo brings up items which I either did not know about or might not have associated with the item in question.

Consequently, I have decided to promote the current version of the demo as "The FreeBSD Browser" (www.cfcl.com/Meta/md_fb.html). If significant numbers of FreeBSD users decide that the browser is useful, I will have achieved a basic "proof of concept" for Meta's basic notions. Also, both the usage logs and the explicit feedback that these users provide should be very valuable in improving the database, rule sets, and code.

Implementation

The demo is implemented as a Perl-based CGI script, fed by a host of information-gathering filters. BerkeleyDB::Btree is used to store a tied hash of (line-oriented and XML format) metadata; Perl and XML::Simple are used to parse the returned data. The metadata comes from:

  • annotation files
  • hard-coded rules
  • system information

Thus, a link might be displayed because of an annotation file entry, a rule in the CGI script, or a "FILES" entry in a man page. To minimize the pain of waiting for results, the CGI script uses a simple form of caching; the HTML for (most) displayed pages is saved for future use.

As of mid-February 2001, the demo's code base contained (roughly):

  • 2,500 lines of Perl
  • 20,000 lines of hand-edited XML
  • 1,500,000 lines of generated XML

Although I try to code cleanly and document adequately, the demo's code base is not ready for public viewing. Nor, given my limited resources and large task list, is the code likely to be "released" any time soon. Interested individuals are welcome to contact me, however, if they wish to assist in the project.

Futures

Although the demo's long-term goals are subsumed in those of the Meta Project, some short-term goals are worth mentioning:

  • better tuning of the graph traversal rules
  • graphical display of selected "subsystems"
  • increased coverage and detail in the metadata

If the demo attracts enough users, the current implementation could become strained. Fortunately, several steps can be taken to improve its performance:

  • switch to faster (and perhaps more) hardware
  • supply the server(s) with better bandwidth
  • force production and caching of popular pages
  • recode to use FastCGI, an RDBMS, etc.

In the medium term, I hope to greatly expand the demo's depth and breadth. Online browsing of cross-linked files (e.g., source code and documentation) is quite feasible. Coverage of add-on packages and other operating systems (e.g., BSD, Linux, or even proprietary Unix variants) is also a strong possibility.

-- Main.RichMorin - 16 Jun 2003
Topic revision: r3 - 08 Jun 2003, WikiGuest
This site is powered by Foswiki Copyright © by the contributing authors. All material on this wiki is the property of the contributing authors.
Foswiki version v2.1.6, Release Foswiki-2.1.6, Plugin API version 2.4
Ideas, requests, problems regarding CFCL Wiki? Send us email