Scripting Queries In Ivy

Ivy leverages Kx Systems kdb+ column-store database and q programming language. For over 30 years, this technology has been used by the financial industry to handle real-time trading and is unmatched when it comes to the speed and scalability required for Big Data.

Filtering and rendering in seconds!

With Ivy and kdb+, interactive real-time Big Data analysis becomes a reality for any analyst familiar with SQL. This powerful language lets you perform billion row queries in just a few milliseconds. In the demonstration below, we load up a billion row table and then perform a simple date query to filter the dataset down to 60 million records in 86ms. Then we render the 60 million records as a binned heat map in about 7 seconds. We're using a C3.2xLarge Amazon instance but we are only using 6 of the 8 cores.

 

Analysts can use the Visual Query Builder to load and join tables and perform complex scripting. However, some analysts prefer to interactively and incrementally build their queries in the Big Data Spreadsheet, the Function Editor, or directly in the Visual Inspector. Here are a few simple examples to show you how easy it is to interactively script queries in Ivy.

On line 11, we are selecting all the columns from the phones table where the destination column contains either 1, 5, 306 or 309. On line 13, we are counting all the sources grouped by their destination. Then, we are adding those counts to a column called counts. And finally we are sorting the new counts column in descending order.

On line 17, we are selecting all the records where duration in greater than 1500 and assigning them to a new table called a. On line 18, we are selecting all the records with a destination of either 0 or 5 and assigning them to a new table called b. Then on line 19, we are merging all of the records from a and b into a new table called c.

In this way, an analyst can perform a variety of ad-hoc queries on-the-fly and inspect the result of each of those queries without having to build up a massive SQL statement. This interactive flexibility makes it faster and easier to explore large datasets. And since any selection of lines can be inspected, analysts always know where they stand with their data.

In addition to the enhanced query language, programmers, data scientists and analysts can also leverage q -- a powerful vector functional programming language. q is a full-blown programming language that gives analysts complete control over their environment.