Data Flow

In the simplest terms, Meta's data flow can be shown as a "pipeline", flowing from ExistingSources and HumanEffort, through a series of processing stages, to the SupportedSoftware:

  --------------------   ----------------
  | Existing Sources |   | Human Effort |
  --------------------   ----------------
     | | | | | | | |       | | | | | |      <- random data (random formats :-)
  ---------------------------------------
  |    Filter Engine + Input Filters    |
  ---------------------------------------
                     |||                   <- Characterizations & Conversions (XML)
  ---------------------------------------
  |       Index Engine + Indexers       |
  ---------------------------------------
                     |||                   <- ... + indexes (XML)
  ---------------------------------------
  |      Analysis Engine + Analysts     |
  ---------------------------------------
                     |||                   <- ... + conclusions (XML)
  ---------------------------------------
  |         Aggregator/Selector         |
  ---------------------------------------
                     |||                   <- "Publishable" database (XML)
  ---------------------------------------
  |    Format Engine + Output Filters   |
  ---------------------------------------
        |||  |||  |||  |||  |||  |||       <- ... (assorted formats)
  ---------------------------------------
  |         Selector/Distributor        |
  ---------------------------------------
        |||  |||  |||  |||  |||  |||       <- database extracts, as desired
  ---------------------------------------
  |          Supported Software         |
  ---------------------------------------

This model neglects, to be sure, the feedback loops which allow the system to improve both its input data and processing strategies. For instance:

  • A browser might accept feedback from the user and send it back to the Meta Project.

  • Meta (and its users) may send error reports and enhancement requests to the Existing Sources.

  • As deficiencies are noticed (and time allows :-), improvements can be made to Meta's processing code.

So, think of this as an "instantaneous" snapshot of the data flow, neglecting larger-scale system effects.

RDBMS Notes

As an alternative to creating a directory tree of XML files, we are considering using a Relational Data Base System (RDBMS) to store the server-side metadata. XML versions of any desired metadata could, of course, be generated at any stage. For a discussion of this idea, see Why RDBMS and Query Example.

-- Main.RichMorin - 16 Jun 2003
Topic revision: r4 - 08 Jun 2003, WikiGuest
This site is powered by Foswiki Copyright © by the contributing authors. All material on this wiki is the property of the contributing authors.
Foswiki version v2.1.6, Release Foswiki-2.1.6, Plugin API version 2.4
Ideas, requests, problems regarding CFCL Wiki? Send us email