Universe

Join the chat at https://gitter.im/crossfilter/universe

Build Status Code Climate

The easiest and fastest way to explore your data

Before Universe, exploring and filtering large datasets in javascript meant constant data looping, complicated indexing, and countless lines of code to dissect your data.

With Universe, you can be there in just a few lines of code. You've got better things to do than write intense map-reduce functions or learn the intricate inner-workings of Crossfilter ;)

Features

Features in the Pipeline

Demos

API

Getting Started

Installation

NPM

npm install universe --save

Download from the releases page. Serve the universe.js or universe.min.js file in the top-level directory as part of your application.

Create a new Universe

Pass universe an array of objects or a Crossfilter instance:

var universe = require('universe')

var myUniverse = universe([
    {date: "2011-11-14T16:17:54Z", quantity: 2, total: 190, tip: 100, type: "tab", productIDs: ["001"]},
    {date: "2011-11-14T16:20:19Z", quantity: 2, total: 190, tip: 100, type: "tab", productIDs: ["001",  "005"]},
    {date: "2011-11-14T16:28:54Z", quantity: 1, total: 300, tip: 200, type: "visa", productIDs: ["004", "005"]},
    ...
  ])
  .then(function(myUniverse){
    // And now you're ready to query! :)
    return myUniverse
  })

Query your data

.then(function(myUniverse){
  myUniverse.query({
    groupBy: 'type' // GroupBy the type key
    columns: {
      $count: true, // Count the number of records
      quantity: { // Create a custom 'quantity' column
        $sum: 'quantity' // Sum the quantity column
      },
    },
    // Limit selection to rows where quantity is greater than 50
    filter: {
      quantity: {
        $gt: 50
      }
    },
  })

  // Optionally post-aggregate your data
  // Reduce all results after 5 to a single result using sums
  myUniverse.squash(5, null, {
    count: '$sum',
    quantity: {
      sum: '$sum'
    }
  })

  // See Post-Aggregations for more information
})

Use your data

.then(function(res) {
  // Use your data for tables, charts, data visualiztion, etc.
  res.data === [
    {"key": "cash","value": {"count": 2,"quantity": {"sum": 3}}},
    {"key": "tab","value": {"count": 8,"quantity": {"sum": 16}}},
    {"key": "visa","value": {"count": 2,"quantity": {"sum": 2}}}
  ]

  // Or plost the data in DC.js using the underlying crossfilter dimension and group
  dc.pieChart('#chart')
    .dimension(res.dimension)
    .group(res.group)

  // Pass the query's universe instance to keep chaining
  return res.universe
})

Explore your data

As you filter your data on the universe level, every query's result is updated in real-time to reflect changes in aggregation

// Filter records where 'type' === 'visa'
.then(function(myUniverse) {
  return myUniverse.filter('type', 'visa')
})

// Filter records where 'type' === 'visa' or 'tab'
.then(function(myUniverse) {
  return myUniverse.filter('type', ['visa', 'tab'])
})

// Filter records where 'total' is between 50 and 100
.then(function(myUniverse) {
  return myUniverse.filter('total', [50, 10], true)
})

// Filter records using an expressive and JSON friendly query syntax
.then(function(myUniverse) {
  return myUniverse.filter('total', {
    $lt: { // Filter to results where total is less than
      '$get(total)': { // the "total" property from
        '$nthLast(3)': { // the 3rd to the last row from
          $column: 'date' // the dataset sorted by the date column
        }
      }
    }
  })
})

// Or if you're feeling powerful, just write your own custom filter function
.then(function(myUniverse){
  return myUniverse.filter({
    total: function(row){
      return (row.quantity * row.sum) > 50
    }
  })
})

// Clear the filters for the 'type' column
.then(function(myUniverse){
  return myUniverse.filter('type')
})

// Clear all of the filters
.then(function(myUniverse){
  return myUniverse.filterAll()
})

Clean Up

// Remove a column index
.then(function(myUniverse){
  return myUniverse.clear('total')
})

// Remove all columns
.then(function(myUniverse){
  return myUniverse.clear()
})

API #

universe( [data] , {config} ) #

.query( {queryObject} ) #

.filter( columnKey, filterObject, isArray, replace ) #

.column( columnKey/columnObject ) #

.clear( columnKey/columnObject/[columnKeys/columnObjects] ) #

.add( [data] ) #

Post Aggregation #

Post aggregation methods can be run on query results to further modify your data. Just like queries, the results magically and instantly respond to filtering.

Locking a query

A majority of the time, you're probably only interested in the end result of a query chain. For this reason, Post Aggregations default to mutating the data of their direct parent (unless the parent is the original query), thereby avoiding unnecessary copying of data. On the other hand, if you plan on accessing data at any point in the middle of a query chain, you will need to lock() that query's results. This ensure's it won't be overwritten or mutated by any further post aggregation.

Note: Running more than 1 post aggregation on a query will automatically lock the parent query.

.then(function(universe){
  return universe.query({
    groupBy: 'tag'
  })
})
.then(function(query){
  query.lock()
  var all = query.data
  return query.limit(5)
})
.then(function(query){
  var only5 = query.data

  all.length === 10
  only5.length === 5
})

Without locking the above query before using .limit(5), the all data array would have been mutated by .limit(5)

.sortByKey(descending) #

.limit(n, n2) #

.squash(n, n2, aggregationMap, keyName) #

.change(n, n2, changeFields) #

.changeMap(changeMapObj) #

.post(callback) #

Pro Tips #

No Arrays Necessary

Don’t want to use arrays in your aggregations? No problem, because this:

.then(function(universe){
  universe.query({
    select: {
      $sum: {
        $sum: [
          {$max: ['tip', 'total’]},
          {$min: ['quantity', 'total’]}
        ]
      },
    }
  })
})

… is now easier written like this:

.then(function(universe){
  universe.query({
    select: {
      $sum: {
        $sum: {
          $max: ['tip', 'total'],
          $min: ['quantity', 'total']
        }
      },
    }
  })
})

No Objects Necessary, either!

What’s that? Don’t like the verbosity of objects or arrays? Use the new string syntax!

.then(function(universe){
  universe.query({
    select: {
      $sum: '$sum($max(tip,total), $min(quantity,total))'
    }
  })
})

Pre-compile Columns

Pro-Tip: You can also pre-compile column indices before querying. Otherwise, ad-hoc indices are created and managed automagically for you anyway.

.then(function(myUniverse){
  return myUniverse.column('a')
  return myUniverse.column(['a', 'b', 'c'])
  return myUniverse.column({
    key: 'd',
    type: 'string' // override automatic type detection
  })
})