Architecture

Some parts of the Ontiki architecture are pretty clear, but other parts are still rather murky:

  • Ontiki will use a web-based client / server architecture and a graph database.

  • For security, all incoming requests will be handled by the server, Ontiki_S.

Client Application

The client will be a browser-resident single-page application. It will handle presentation and user interaction, using assorted nifty libraries, eg:

  • 2D data visualization (D3, ...)

  • FRP-based event handling (Elm, Om, ...)

Server Application

At least initially, the server will use a Ruby-based Web Framework (e.g., Rack, Roda, Ruby on Rails, Sinatra). This can be reimplemented in another language later, if need be. The main data store will use Neo4j (a graph database).

Diagrams

Components

Here's a high-level look at Ontiki's components, including dependencies (e.g., applications, frameworks, libraries, servers):

Client / Server Division

The diagrams below show Ontiki's client and server app suites, separated by a dashed, horizontal line. This emphasizes the fact that all incoming requests are handled by Ontiki_S. The right-hand diagram adds a Gremlin (i.e., TinkerPop3) server for graph manipulation (and possibly for some kinds of Neo4j access):

   

Ontiki_S may also talk to ancillary servers (e.g., Cache, Logs). To keep the diagram clean, let's just show these as independent "listeners":

Discussion

Cache, Logs, etc.

Ontiki_S will be sending and receiving lots of messages (e.g., to other servers). Caching might be a win for external servers; logging would aid debugging and analysis in all cases. So, why not cache the external traffic and log the rest? (Disk is free, after all. :-) Fluentd and Git may play useful roles here, but I'm not clear about the details yet...

Gremlin

Gremlin (i.e., TinkerPop3) integrates a wide range of graph manipulation tools into a single, vendor-independent suite. Also, its property graph model looks a lot like Neo4j's. Unfortunately (from my perspective):

  • It's based on Java, so it can't run in the client.

    This is a deal-killer for real-time feedback.
    It also makes a lot more work for the server.

  • It uses imperative (Groovy) descriptions of traversals.

    However, it's possible to embed Cypher code,
    making possible some sort of a creole DSL.

  • It's sending executable code to a Server (yeeps!).

  • It represents a steep learning curve on a moving target.

    As of 2014/12, it's at version 3.0.0M3, with no book.

So, I plan to keep an eye on it, but approach it very cautiously, eg:

  • Get the Neo4j-based setup working.

  • Set up an instance of TinkerPop3.

  • Compare query results, timing, etc.

HTTP Proxy

It isn't necessary to run Ontiki_S on the site's main machine. Apache's mod_proxy will pass requests to other (local) machines:

Support Applications

These apps have to do with data importing, graph analysis, and other tasks that help to support Ontiki.

Discovery

It would be interesting (and potentially very useful) to run background apps that attempt to discover relationships and patterns in the database. See Discovery for details.

RDF Import

A lot of interesting content (e.g., YAGO) is distributed as RDF triples, typically encoded as Turtle. So, we need a convenient and flexible way to map this onto Neo4j. The current import suite, built from a combination of Ruby scripts and the Unix system sort utility, has some annoying issues:

  • It embodies YAGO's conventions, schema, etc.

  • The Ruby code is slow and single-threaded.

As I bring in other data sets, I will generalize the suite. I may also translate the Ruby scripts into a language (eg, D, Elixir, Rust) that performs better and handles concurrency well. See YAGO - Installation and YAGO - Importing for details.


This wiki page is maintained by Rich Morin, an independent consultant specializing in software design, development, and documentation. Please feel free to email comments, inquiries, suggestions, etc!

Topic revision: r13 - 29 Apr 2015, RichMorin
This site is powered by Foswiki Copyright © by the contributing authors. All material on this wiki is the property of the contributing authors.
Foswiki version v2.1.6, Release Foswiki-2.1.6, Plugin API version 2.4
Ideas, requests, problems regarding CFCL Wiki? Send us email