Plumbing

Most of AxAp's use cases require it to access, transform, and present digital content, in order to make it more accessible. To support this activity, AxAp implements an extensible plugboard of data transformations (e.g., HTML augmentation, format conversion). To accomplish this, AxAp uses a variety of "plumbing" mechanisms, e.g.:

  • Channels, Connections, Pipelines, Plugs, ..., ala Phoenix
  • Files, Sockets, TCP/IP, ... , ala Unix

Intra-process

For intra-process plumbing, AxAp borrows the notions of connections, plugs, and pipelines from Phoenix. It stores all information for each connection in a single data structure (basically, a tree of hashes). Each "plug" mines this structure for control parameters and input data. It may also augment the structure, adding log and/or status information, combined and/or transformed data, etc. A "pipeline" defines a named sequence of plugs which can be "piped through" for particular types of requests.

However, Ruby is not Elixir and Sinatra is not Phoenix. So, AxAp has to do without features such as clean namespace control, strong support for concurrency (let alone distribution), Lisp-style macros, etc. Consequently, AxAp's implementation of connections differ in several ways from the original.

Conn Games

To step around Ruby and Sinatra's notions of namespaces and process boundaries, AxAp refers to the connection object in a variety of ways:

  • $conn - global variable, used during setup
  • @conn - class variable, used during requests
  • conn - local variable, used in plug methods

Although the connection object is not immutable, AxAp's conventions achieve a similar effect. They dictate that plugs should only add elements to the connection hash (as opposed to changing or removing anything). This rule prevents most "breaking" changes.

Discussion

Rich Hickey's Spec-ulation keynote (starting at 22:13) provides some theoretical support for this approach. As software grows, he says, the pieces within it can be expected to undergo various types of breaking and growing (non-breaking) changes:

  • Breaking: require more, provide less, redefine names
  • Growing: provide more, require less, resolve issues

Because adding a new hash element provides more information, it typically doesn't break anything that other code depends on. Removing or changing the value of an element, in contrast, may do so. Changing the meaning of a hash element will break any code that depended on the previous definition. Other collections (e.g., arrays, lists) and scalars can be used as leaf nodes, as long as they are treated as immutable values.

Plugs and Pipelines

Phoenix uses Elixir macros to implement plugs and pipelines. We don't have these, so we use Ruby class methods and data structures. For example, routes/epub_area.rb (which sets up routing for the EPUB area) defines and uses the following pipeline:

pl_epub_area_1  = %i[
  Admin.load_user_accounts
  Admin.load_user_settings
  Admin.load_epub_docs
  EPUB_Access.get_epub_paths
]
...
pipeline pl_epub_area_1

The pipeline method simply traverses the list, calling the plug method on each item. Assuming that the pipeline hasn't been halted, plug runs eval on the specified item. It may also calculate and display its duration, etc. Although functions (let alone eval) are less efficient than Elixir macros, performance isn't generally a problem for AxAp code.

Discussion

The use of data structures to define pipelines has some interesting advantages. Because our pipeline definition (e.g., pl_epub_area_1) is just a list of symbols, it can be manipulated (e.g., filtered) at runtime. In addition, pipeline definition lists can be created or modified dynamically, based on runtime requirements.

So, we may be able to generate pipelines from sets of patterns and transformations, ala DIET (described by Dave Thomas in Transforming Programming). Of course, we still have to figure out how to specify the goal (i.e., presentation format) and find a path to it, but that seems doable.

Inter-process

Because the AxAp server can't do everything by itself, it has to communicate with other processes. For example, interacting with web pages and other servers on the Internet requires it to interact with remote processes, Some data transformations will be handled by local processes (e.g., ImageMagick, Pygments). Finally, many concurrent and/or distributed processing tasks can be handled better by Elixir than by Ruby.

Fortunately, we have a wealth of options for inter-process plumbing, provided by Elixir, Erlang, Phoenix, Ruby, Unix, and the Web. We can use channels, files, messages, pipes, sockets, etc. Indeed, defining a minimal subset that serves most of our needs may be the real challenge.

We expect many of AxAp's use cases to be implemented in Elixir, taking advantage of Erlang's BEAM and related infrastructure. Typically, this will be because they require concurrency, but distribution, macros, reliability, and other features may also be involved. So, we'll be sending and receiving lots of messages, e.g. {:ok, 42}.

The main (Ruby/Sinatra) AxAp server will communicate with a local copy of Phoenix (acting as an internal server), as well as other (Internet-based) servers. This will generally use a RESTish HTTP approach.


This wiki page is maintained by Rich Morin, an independent consultant specializing in software design, development, and documentation. Please feel free to email comments, inquiries, suggestions, etc!

Topic revision: r7 - 27 Mar 2017, RichMorin
This site is powered by Foswiki Copyright © by the contributing authors. All material on this wiki is the property of the contributing authors.
Foswiki version v2.1.6, Release Foswiki-2.1.6, Plugin API version 2.4
Ideas, requests, problems regarding CFCL Wiki? Send us email