Ivy: An Interactive Collaborative Analytics Environment

Ivy is an Interactive Collaborative Analytics Environment that allows both programmers and analysts to work with massive datasets in real-time. It combines our expertise in building cutting-edge development environments such as Visual Age and Eclipse with Kx System’s powerful kdb+® technology to deliver an unparalleled ‘Big Data’ analytics experience.

Using Ivy, analysts can query and visualize billions of records as if they are working with a few thousand. Working in real-time, analysts move seamlessly back and forth between compute-intensive queries or calculations and interactive visualizations. With Ivy, analysts are able to tumble massive datasets using a “think, compute, see” paradigm to find meaning and uncover patterns quickly and easily.

Installing Ivy is easy! Simply unzip the installation and run a script file. Quite literally, your analysts can be up and running in under a minute! And since Ivy runs in an HTML 5 web browser, no client installation is required. Simply give your analysts the URL and they are ready to go.

Ivy-At-A-Glance

The following diagram provides a high-level overview of Ivy and is described in detail below:

Ivy Architecture Chart

Client-Side Browser-Based Tools

Users connect to the Ivy server using a standard HTML5-compliant web browser. A variety of tools are provided for both the data scientist/analyst and the programmer. These tools have been specifically designed for Big Data in terms of speed and scalability.

Analysts can perform ad-hoc, non-linear analysis using the Big Data Spreadsheet and an optimized SQL-like query language or a full-blown vector functional programming language. The Big Data Visual Inspector lets them to discover patterns in their data by looking at billions of rows in real-time in a variety of different chart formats. With the Big Data Visual Query Builder, they can load tables, join tables and build simple or complex SQL queries. In many respects, it is similar to MS-Query but has been optimized for huge in-memory or disk-based datasets. And analysts can import and groom datasets without programming using the Data Importer and Transformer.

For developers, Ivy provides an Interactive Function Editor to assist you with building domain-specific function libraries as well as an Interactive Diff/Merge Tool and Interactive Debugger to simplify and expedite the development process.

Server-Side Shared Repository with Fine-Grain Version Management

Both developers and analysts benefit from using Ivy’s GIT-like shared repository and fine-grained version control. For developers, Ivy adds modules and configurations to the powerful q environment, as well as version control for functions and spreadsheet cells. Compare, diff and merge are all supported. For analysts, Ivy provides a collaborative, shared work experience and analysis history. Analysts work in their own workspace, then share their thoughts and findings with other analysts by publishing part or all of their workspace. In turn, other analysts can comment or make modifications to the ongoing analysis. At any point, anyone can move forward or backward through the versions that have been placed in the repository. In the example below, Ivy artifacts are available in local and shared views. Users can search either the local or shared repository. They can also push items to or pull items from the repository.

Example of shared and local repository tree structure

In-Memory or On-Disk Column Store Technology

Extremely fast queries can be made against either in-memory or on-disk databases. For example, a billion record log file query was reduced to 60 million records in 86ms. The Ivy instance was running on a standard C3.2xLarge Amazon server using only 6 of 8 cores and 15GBs of memory. In another example, a trillion record stock trade and quote query was reduced to 35 million records in 18ms. Tables can be stored in a variety of formats for fast on-disk query (serialized, splayed, partitioned/sharded, segmented).

High-Performance Vector Functional Programming Language
and Enhanced High-Speed SQL Query Language

Ivy leverages Kx Systems kdb+ column-store database and q programming language. The language features an enhanced SQL-like query language that allows you to incrementally develop table join, query and filter operations. An example of the query language is shown below. For over 30 years, this technology has been used by the financial industry to handle real-time trading and is unmatched when it comes to the speed and scalability required for Big Data. In fact, we were so impressed with the technology, we used it to build Ivy!

Example of query language

Import Formats, Output Formats and Supported Interfaces

Ivy doesn’t lock you in. Users can import datasets using CSV, JSON, XML and ODBC/JDBC interfaces. Self-serve helper wizards are provided to simplify the import and transform process. Once your data is in Ivy, you can save it in a wide range of formats such as column store, partitioned data,  import functions or helper applications; create your own libraries using kdb+ and q; create custom visualizations using Ivy’s server-side visualization API or any other client-side library (e.g. D3); and interface to wide range of external libraries/languages/applications (Java, C, C++, C#, R, Python, Perl, Scala, Excel, Fortran, ODBC/JDBC).