How and where to use Data Turbine
Submitted by hubbard on Wed, 04/30/2008 - 16:54.
Distributed
Heterogenous
Collaborative
Let me explain what I mean.
Since DataTurbine adds a flexible-length buffer and access to your design, it adds capabilities like TiVo-style browsing, replay, rewinding, and more. However, if everything is simple, local, and point-to-point then perhaps DataTurbine might not be helpful.
Since its a TCP/IP-based network application, it makes a wonderful abstraction layer for heterogenous devices; once data is sent to DataTurbine the underlying hardware and software are totally abstracted. The clients don't need to know or care. However, if you just have one device type, then abstraction is less useful.
I posit that DataTurbine is most useful when you're collaborating on science, especially with remote users. It lets anyone (who you permit) view data, watch video, annotate an ongoing experiment and zip through data with ease. All of those capabilities are great for a single lab, but they're superb when, say, you want to view the experiments' progress from your iPhone, or let your remote collaborator control the mass spec from their office.
DataTurbine's ability to mix and match numbers and video, all synchronized, is quite valuable. You can not only see the video feed, but you also see what was happening on the other sensors or feeds when it happened. Automatically, all the time, no extra effort required.
The Network for Earthquake Engineering and Simulation(NEES) uses it for distributed experiments, often spanning thousands of miles and several days. For example, the structural lab at UMN collaborates with researchers in Puerto Rico, who can observe, annotate and control without frequent flyer miles to Minnesota. UMN uses LabWindows-based data acquisition systems, Axis network cameras, streaming audio servers (custom code based around Windows and the Java Media Framework), as well as Java-controlled consumer digital cameras on remotely-controlled imaging towers.
We're using DataTurbine to demonstrate streaming numeric & video datafrom the Santa Margarita Ecological Reserve and a coral reef in Taiwan. This uses LabVIEW CompactRIO data acquisition, a variety of sensors, numerous network video systems and at least four different operating systems, and that's before you include the network gear!
Insight Racing is using it for video onboard their autonomous car. Video from Axis 206M 1.3 megapixel cameras streams to an onboard Linux-based video server for processing over Ethernet.
NASA uses RBNB for telemetry as part of their "Intelligent Network Data Server" or INDS at their Dryden Lab.
Here are some more ideas for places where DataTurbine is a useful capability:
Stream processing and event detection. You can easily interface to DataTurbine from Matlab, givng you the ability to write programs that process and analyze streams of data in real time. Think video processing, complex algorithmic event detection, anything you can do with the vast cornucopia of Matlab code or toolboxes. Output can be sent right back to DataTurbine as another data stream, viewable by anything in the system.
Event markers and annotation. We have a capability to annotate the DataTurbine that we call 'Event markers.' These are timestamped bits of XML, stored in the DataTurbine as a text channel and part of the record. You can use them as notes or annotation, either human or programmatic, and are great for situational knowledge or just understanding an experiment afterwards. For example, 'hydraulics now online,' 'sensor X on box Y is faulted and offline' or 'I think we just saw beam failure on cantilever 3' are all examples of possible markers. These can be generated and viewed in RDV or your own interface.
I hope this has helped give you some ideas of where DataTurbine wins. Please feel free to jump onto the mailing list and ask questions or tell us what you're up to. DataTurbine is under the Apache 2.0 license, but we'd appreciate greatly the courtesy of an email telling us how you're using it, any feedback or features you'd like to see.
If you're new to DataTurbine, you're perhaps wondering how and where it's useful, and if it'd be useful for your own work. This document aims to answer those questions for you, and give you some ideas.
Before I dive into what it can do, let's focus a bit on systems you might have encountered, so that you know what's different.
How is DataTurbine different than other systems:
- The 'network ring buffer' idea lets you separate data sources, and also lets you mix and match different rates. Unlike an Enterprise Message Service (EMS) system, clients can get old or new data, as required, and can look at data more than once.
- Unlike a database, you can subscribe to a data feed, and get efficient, low-latency propagation with high throughput (tens of megabytes per second, tens to hundreds of clients)
- Unlike a filesystem, there are no locks to worry about, byte ordering issues, but you can still random-access address the data streams.
- Unlike a simple TCP or multicast connection, you can have different rates. Sources can 'burst' data, that clients can let pile up and process as able and interested. Clients can also choose which sources to monitor, so they needn't waste bandwidth seeing everything.
Since DataTurbine is Java-based, it runs anywhere there's a JVM, from Gumstix to 64-bit servers. If your system can't run Java, you can still stream data using a proxy (several are provided, contact us for details) or via DataTurbine's WebDAV filesystem interface. Our best display client is called RDV, and its a rich-client Java application. There are web interfaces for data, but none are as capable, so in general Java is useful-to-essential for serious use of DataTurbine.
DataTurbine is more useful in applications that are
Let me explain what I mean.
Here are some examples where DataTurbine is a big win:
Here are some more ideas for places where DataTurbine is a useful capability:
I hope this has helped give you some ideas of where DataTurbine wins. Please feel free to jump onto the mailing list and ask questions or tell us what you're up to. DataTurbine is under the Apache 2.0 license, but we'd appreciate greatly the courtesy of an email telling us how you're using it, any feedback or features you'd like to see.
Data Turbine is a commercial
Data Turbine is a commercial Java application produced by Creare, Inc. external link icon Also known as RBNB (Ring Buffered Network Bus) cheap bulletin hosting, Data Turbine provides high performance time-synchronized data streaming services for both static and dynamic budget hosting data. NEESit’s Data Turbine utilities provide an interface between a Data Turbine server and connected sources and sinks that provide and request data, respectively business web hosting. This document describes how to use NEESit’s Data Turbine utilities. It also describes how to obtain the source code, which includes two small demonstration programs that serve as a tutorial for programmers wishing to create their own Data Turbine utilities ecommerce web hosting. Information about how to install these utilities is located in the Getting Started With Data Turbine external link icon guide.