DataTurbine Frequently Asked Questions

This is the FAQ book (set of pages) for RBNB DataTurbine. The idea is to add a new child page for each topic, so that this is an easily-maintained living document.

Please see this Drupal doc page for the mechanism.

Administering DataTurbine

This assumes that DataTurbine has been deployed as described in the FAQ.

- The DataTurbine server program is contained in the executable Java Archive (jar),

${RBNB_HOME}/bin/rbnb.jar. 

Invocations of this program take the form:

java -jar ${RBNB_HOME}/bin/rbnb.jar 
(in Windows terminal, it's java -jar %RBNB_HOME%\bin\rbnb.jar)

with command-line arguments:

  -a {server address - [host][:port]}
  -A {security file URL}
  -D {mask},{level}
  -F
  -H {archive home directory}
  -l [{log period (millis)}][,{log cache frames}][,[{log cache frames}][,[{log archive frames}][,[{log archive mode}]]]]
  -L
  -m [{metrics period (millis)}][,[{metrics cache frames}][,[{metrics archive frames}][,[{metrics archive mode}]]]]
  -M {maximum activity threads}
  -n {server name}
  -O {options file URL}
  -p {parent server address}
  -P [{userid}][,{password}]
  -s {shortcut name},{remote address},[{cost}]
  -S {maximum open filesets}

SOFTWARE ADMINISTRATION

- Select filesystem locations where the DataTurbine server is to maintain its disk archive and log. These locations will be referred to as RBNB_ARCHIVE and RBNB_LOG, respectively, from this point on. The preparation of convenience scripts to control the server is highly recommended. Some examples are at the end of this article.
- DataTurbgine ships with an administration utility, ${RBNB_HOME}/bin/admin.jar. When connected to a DataTurbine server, it displays a list of all entities available from that server. This is handy as a very basic verification of the presence of sources and channels. It also provides actions via a right-click menu popup. Some particularly useful features are:
- Terminate - removes a source from the servers cache and commits its data to the disk archive (if there is one).
- Load Archive - loads the disk archive from an inactive source into the cache. This requires prior knowledge of the archive's name.

EXAMPLES
- Linux (make a file called rbnb; this is a proper /etc/init.d script)

#!/bin/bash
#
# Init script for the RBNB server
#
# chkconfig: 345 98 98
# description: Start the RBNB server
#
# Megabytes to use for the JVM running RBNB
RBNB_MEM='512'
RBNB_ARCHIVE='/var/rbnb_archive'
RBNB_LOG ='/var/log/rbnb.log'
RBNB_PIDFILE='/var/run/rbnb.pid'

start() {
	mkdir -p ${RBNB_ARCHIVE}
  	java -Xmx${RBNB_MEM}M -jar ${RBNB_HOME}/bin/rbnb.jar -n dataturbine -L -H ${RBNB_ARCHIVE} > /var/log/rbnb.log 2>&1 &
   echo $! > ${RBNB_PIDFILE}
   }

stop() {
 	kill `cat ${RBNB_PIDFILE}`
 	rm -f ${RBNB_PIDFILE}
}
     
case "$1" in
   'start')
       echo "Starting the RBNB server"
       start
       ;;
   'stop')
       echo "Stopping the RBNB server"
       stop
       ;;
   'restart')
       echo "Restarting the RBNB server"
       stop
       start
       ;;
   *)
       echo "Usage: ${0} {start|stop|restart} "
       ;;
esac

- Windows (make a file called rbnb.bat)

set RBNB_ARCHIVE=C:\\rbnb_archive
set RBNB_LOG=rbnb.log
mkdir %RBNB_ARCHIVE%

java -jar %RBNB_HOME%\\bin\\rbnb.jar -L -H %RBNB_ARCHIVE% -n "dataturbine" >> %RBNB_LOG% 2>&1

SunT2000

Information and notes about our Sun T2000 server, niagara.sdsc.edu

Network information:
ipge3 is niagara-stage.sdsc.edu
IP address: 137.110.118.233
your netmask is: 255.255.255.0
your default gateway address is: 137.110.118.1
your primary nameserver address is: 132.239.1.52
your secondary nameserver address is: 128.54.16.2

ipge2 is niagara-dev.sdsc.edu
IP address: 137.110.118.234
your netmask is: 255.255.255.0
your default gateway address is: 137.110.118.1

ipge1 is unnamed, VLAN-bridged to the NEES internal network.
192.168.0.137 netmask 255.255.255.0 (non-routable)

ipge0 is niagara.sdsc.edu
137.110.118.159

Bridging Antelope and DataTurbine

A DataTurbine source program that effectively mirrors channels and numeric data streams from an Atelope Object Ring Buffer into a DataTurbine Ring Buffer can be found here:
http://code.google.com/p/oss-dataturbine/source/browse/trunk/apps/oss-apps/src/org/nees/rbnb/Orb2Rbnb.java
http://code.google.com/p/oss-dataturbine/source/browse/trunk/apps/oss-apps/src/org/nees/rbnb/Orb2RbnbTTask.java

This program has a Java implementation of the Antelope API contributed to the BRTT Antelope users group as a prerequisite.

Can I mix and match different dataloggers/cameras/DAQs in a single DataTurbine?

Yes! It's designed for just that. Make sure that the clocks are synchronized, and that's about it.

DataTurbine metrics channels

Built-in RBNB _Metrics channels can be viewed using RDV or rbnbPlot. In particular watch MemoryUsed, TotalMemory, and SocketRate. The first two will let you see when Java garbage collection events occur which are often a big culprit for unexpected system load. The latter will give you feedback about overall client activity. E.g. with no sinks active you will see the baseline data flow for your sources. Beware of a Heisenberg-like uncertainty principle; the act of fetching _Metrics to look at them will change the _Metrics. Also beware sinks that seem inactive but may actually be status polling etc.

DataTurbine programming gotchas and tips

  1. DataTurbine is divided into Sources, Sinks and Plugins.
    1. Sources generate data (writers)
    2. Sinks consume data (readers)
    3. Plugins operate on data and produce derived output, similar to a Unix pipe.
  2. Names in the turbine are hierarchical. The simplest is Source/Channel where a source can have more than one channel, but a channel belongs to exactly one source.
    1. If you have parent-child routing, you will see Parent/Child/Source/Channel
    2. Loops in the graph are very possible due to shortcuts and mirroring. Tree traversal algorithms should expect and deal with cycles.
  3. If you connect with a duplicate Source name, DataTurbine will silently give you an auto-incremented name, e.g. Source_1. However, if you specify "append" when connecting, and you are the only instance, it'll reconnect to the old data and append.
  4. All sources should call Detach before Close, otherwise your data will vanish when you disconnect. This loss is rarely the desired behaviour.
  5. Server names can be a TCP name or user-defined from the command line via the -n argument.
  6. Sources can be in several native types - INT16/32, float, double, string or blob.
    1. Numeric data is usually stored as float or double (e.g. instrument data)
    2. Audio data, currently experimental, is stored as INT16
    3. Event markers are XML, stored as string type
    4. Video is actually discrete JPGs, stored as binary blobs, one per image
  7. Time synchronization is critical - all machines must be NTP-synced. That includes the people running data viewers like RDV! See this writeup for more details.
  8. Timestamps are stored as doubles, with 32 bits for integer time_t and 32 bits for fractional seconds.
  9. RDV can't display data sampled faster than 1khz at present.
  10. Metadata in DataTurbine is split into two types.
    1. Invariant. This are things that are set once and never change, for example units. These are set using the PutUserInfo call, where the contents are "name=value,name2=value2..." pairs in a string.
    2. Time-varying. These are stored as normal source feeds. An example would be GPS position of a datalogger.
  11. Each and every source defines its own cache parameters. When a source connects, you specify cache and archive size. This allows you to tune server usage, source by source.
  12. Source data is aggregated into 'frames', which are pushed to the server via the Flush call. You can aggregate (buffer) data to make larger packets, reduce server/network load and increase efficiency.
  13. DataTurbine exposes its internal metrics as 'hidden' channels in the _Metrics folder - memory usage, bandwidth, disk used and more. Very useful.
  14. It also sends its logfiles out as hidden text channels in the _Logs source.
  15. Think of DataTurbine as a very robust abstraction layer: Once data is sent, no sink need worry what kind of device it came from, or how it got there. Enforced device abstaction, network transparency and more!
  16. The DataTurbine server has IP-level access control (read, write) similar to /etc/hosts.deny. There's also a currently-unused mechanism for requiring passwords for sources, but this really needs encryption to be useful and secure.
  17. Getting data into DataTurbine is often the easy part. Once there, you need a good viewer that lets users interact with the data in ways that they find useful.
  18. There are many clients (sinks) as well as DataTurbine->SQL code, file writers, etc so you can use existing tools.
  19. The ChannelMap.PutDataAsXXXX calls do not copy the data. They just save a reference to it, so be sure to leave the data in a valid variable until you call Flush. Otherwise multiple channels will have the exact same value, very puzzling.

DataTurbine Security Architecture

Plese refer to the following document for an overview of Open Source DataTurbine Security architecture [pdf].

Deploying DataTurbine software

SYSTEM PREREQUISITES
- Computer platform capable of running a Java Virtual Machine (JVM) and possessing resources of memory and storage that can accommodate required volume of data
- Java Virtual Machine, version 1.3 or later

SOFTWARE INSTALLATION
- Download the desired version of the DataTurbine installation package from here:
http://code.google.com/p/dataturbine/
- run the installer, which copies files to a specified filesystem location

SOFTWARE CONFIGURATION
- Define the environment variable RBNB_HOME to refer to the root of the filesystem location into which DataTurbine has been installed. Some examples of doing this at the system level are:
- Windows:
start->control panel->system->advanced->environment variables
choose to define RBNB_HOME at the system level
- Linux:
place a file called "rbnb_home" into /etc/profile.d that contains:
export RBNB_HOME='path_to_rbnb'

SOFTWARE VERIFICATION
- "Black Box" verification of a real-time data stream can be achieved by generating a data stream and then viewing this stream with a third-party data viewer. A convenient source of data can be generated with a program that is provided in the DataTurbine installation:
${RBNB_HOME}/bin/source.jar
Running this program will provide a GUI panel that can be use to generate different periodic waveforms.
- Do "File->Open Server" to connect to the DataTurbine server to be verified.
- Verify this real-time stream with a third-party client called RDV, which can be downloaded here:
http://sourceforge.net/projects/rdv/
- Do "File->Connect"; CHANGE the port to 3333
- Double-click channels on the left-hand list to open time series displays
- Put RDV in real-time mode to verify streaming of wave-forms

Enabling and using the PNG plugin for static display of numeric streams

Simple web interface to DataTurbine?

One question is "How do I display data in a web browser? The simplest answer to that is to use the PNG plugin, which renders numeric data into graphs on request. (You can also create a Java applet that runs in the browser, we've had mixed results with that approach)

This is a nifty Creare-supplied turbine plugin that takes data out of the turbine, plots it and outputs the result as a static PNG image. We're going to use it to look at Fake DAQ and some internal server metrics.

The plugin is worth remembering. It's an easy way to get a snapshot of any numeric feed in the turbine, which otherwise requires interpreting a IEEE float; ug.

Load up the turbine web interface in a new window. Make sure that you are on the same machine! The method they use to run programs and plugins only works on localhost! If you're remote, it fails silently. If I need to do this, I usually open an X session to the server and run Mozilla via X11.

Scroll down to Plugins and select "Portable Network Graphic PlugIn - Converts time series data to a PNG format plot, viewable via most Web browsers." You should see a page that's titled 'Portable Network Graphic Plugin'

For now, we'll take the defaults. Note, however, that this is where you set the image resolution. If you want bigger pictures, this is where you change the size. Hit 'Start Plugin'.

A quick check to see if everything is working

Load up this page, and you should see something like a 'directory listing' of the channels on the DataTurbine.

Now what?

Sample images and URLs to come once I figure out how to upload in-line images to dataturbine.org

How and where to use Data Turbine

If you're new to DataTurbine, you're perhaps wondering how and where it's useful, and if it'd be useful for your own work. This document aims to answer those questions for you, and give you some ideas.

Before I dive into what it can do, let's focus a bit on systems you might have encountered, so that you know what's different.

How is DataTurbine different than other systems:

  • The 'network ring buffer' idea lets you separate data sources, and also lets you mix and match different rates. Unlike an Enterprise Message Service (EMS) system, clients can get old or new data, as required, and can look at data more than once.
  • Unlike a database, you can subscribe to a data feed, and get efficient, low-latency propagation with high throughput (tens of megabytes per second, tens to hundreds of clients)
  • Unlike a filesystem, there are no locks to worry about, byte ordering issues, but you can still random-access address the data streams.
  • Unlike a simple TCP or multicast connection, you can have different rates. Sources can 'burst' data, that clients can let pile up and process as able and interested. Clients can also choose which sources to monitor, so they needn't waste bandwidth seeing everything.

Since DataTurbine is Java-based, it runs anywhere there's a JVM, from Gumstix to 64-bit servers. If your system can't run Java, you can still stream data using a proxy (several are provided, contact us for details) or via DataTurbine's WebDAV filesystem interface. Our best display client is called RDV, and its a rich-client Java application. There are web interfaces for data, but none are as capable, so in general Java is useful-to-essential for serious use of DataTurbine.

DataTurbine is more useful in applications that are

  • Distributed
  • Heterogenous
  • Collaborative

    Let me explain what I mean.

  • Since DataTurbine adds a flexible-length buffer and access to your design, it adds capabilities like TiVo-style browsing, replay, rewinding, and more. However, if everything is simple, local, and point-to-point then perhaps DataTurbine might not be helpful.
  • Since its a TCP/IP-based network application, it makes a wonderful abstraction layer for heterogenous devices; once data is sent to DataTurbine the underlying hardware and software are totally abstracted. The clients don't need to know or care. However, if you just have one device type, then abstraction is less useful.
  • I posit that DataTurbine is most useful when you're collaborating on science, especially with remote users. It lets anyone (who you permit) view data, watch video, annotate an ongoing experiment and zip through data with ease. All of those capabilities are great for a single lab, but they're superb when, say, you want to view the experiments' progress from your iPhone, or let your remote collaborator control the mass spec from their office.
  • DataTurbine's ability to mix and match numbers and video, all synchronized, is quite valuable. You can not only see the video feed, but you also see what was happening on the other sensors or feeds when it happened. Automatically, all the time, no extra effort required.

    Here are some examples where DataTurbine is a big win:

  • The Network for Earthquake Engineering and Simulation(NEES) uses it for distributed experiments, often spanning thousands of miles and several days. For example, the structural lab at UMN collaborates with researchers in Puerto Rico, who can observe, annotate and control without frequent flyer miles to Minnesota. UMN uses LabWindows-based data acquisition systems, Axis network cameras, streaming audio servers (custom code based around Windows and the Java Media Framework), as well as Java-controlled consumer digital cameras on remotely-controlled imaging towers.
  • We're using DataTurbine to demonstrate streaming numeric & video datafrom the Santa Margarita Ecological Reserve and a coral reef in Taiwan. This uses LabVIEW CompactRIO data acquisition, a variety of sensors, numerous network video systems and at least four different operating systems, and that's before you include the network gear!
  • Insight Racing is using it for video onboard their autonomous car. Video from Axis 206M 1.3 megapixel cameras streams to an onboard Linux-based video server for processing over Ethernet.
  • NASA uses RBNB for telemetry as part of their "Intelligent Network Data Server" or INDS at their Dryden Lab.

    Here are some more ideas for places where DataTurbine is a useful capability:

  • Stream processing and event detection. You can easily interface to DataTurbine from Matlab, givng you the ability to write programs that process and analyze streams of data in real time. Think video processing, complex algorithmic event detection, anything you can do with the vast cornucopia of Matlab code or toolboxes. Output can be sent right back to DataTurbine as another data stream, viewable by anything in the system.
  • Event markers and annotation. We have a capability to annotate the DataTurbine that we call 'Event markers.' These are timestamped bits of XML, stored in the DataTurbine as a text channel and part of the record. You can use them as notes or annotation, either human or programmatic, and are great for situational knowledge or just understanding an experiment afterwards. For example, 'hydraulics now online,' 'sensor X on box Y is faulted and offline' or 'I think we just saw beam failure on cantilever 3' are all examples of possible markers. These can be generated and viewed in RDV or your own interface.

    I hope this has helped give you some ideas of where DataTurbine wins. Please feel free to jump onto the mailing list and ask questions or tell us what you're up to. DataTurbine is under the Apache 2.0 license, but we'd appreciate greatly the courtesy of an email telling us how you're using it, any feedback or features you'd like to see.

  • How to enable and use the WebDAV filesystem interface

    Configuring Tomcat for WebDAV

    On the newer DT versions, Tomcat ships configured to use port 80. I prefer to change this since a) port 80 is root-only on Unix and b) port 80 often has Apache camped on it already. Note that if either A or B is true, the Tomcat startup script will fail silently.

    To change this, edit

    $RBNB_HOME/apache-tomcat-*/conf/server.xml

    from this

    <Connector className="org.apache.coyote.tomcat4.CoyoteConnector"
    port="80" minProcessors="5" maxProcessors="75"
    enableLookups="true" redirectPort="8443"
    acceptCount="100" debug="0" connectionTimeout="20000"
    useURIValidationHack="false" disableUploadTimeout="true" />

    to this

    <Connector className="org.apache.coyote.tomcat4.CoyoteConnector"
    port="8080" minProcessors="5" maxProcessors="75"
    enableLookups="true" redirectPort="8443"
    acceptCount="100" debug="0" connectionTimeout="20000"
    useURIValidationHack="false" disableUploadTimeout="true" />

    To save your eyes, I've bolded the only part that has to change.

    Enabling write access

    Tomcat is shipped from Creare with the DAV access set to read-only. This is a good security measure, but we'll need to change it. Edit

    $RBNB_HOME/apache-tomcat-*/webapps/webdav/WEB-INF/web.xml

    and change to match this. R/W is just commented out. 

    <!-- Uncomment this to enable read and write access -->
    <!--
    <init-param>
    <param-name>readonly</param-name>
    <param-value>false</param-value>
    </init-param>
    -->

    by removing the comments. You'll have to restart Tomcat for this to take effect.

    How to increase the amount of memory in the DataTurbine

    The DataTurbine's memory is determined by the Java virtual machine's (JVM) startup command-line parameters. These are separate from the RBNB parameters.

    To increase memory, use the

     -Xmx{value}M
    

    parameter. For example:

     java -Xmx1024M -jar $RBNB_HOME/bin/rbnb.jar
    

    would give you 1G of memory.

    Notes and limits

    On a 32-bit operating system, you can't ask for more than about 3500M. On a 64-bit OS like Solaris with a 64-bit JVM, you just add

    -d64
    

    and then increase to what you can afford. We've tested with up to 12G of memory on a Sun T2000.

    LabVIEW support for DataTurbine

    There are several ways to get data from LabVIEW into DataTurbine, depending on your requirements.

    1. Way back in 2002 I wrote LabVIEW code for streaming data on the NEES project. It was designed to be a library of subroutines that you'd add to your code, where streaming was best-effort and I tried hard to make it low-impact. For example, the code will handle network errors transparently, and you have to add a half-dozen routines to your DAQ program. Data is streamed in ASCII over TCP to DaqToRbnb, a Java program in our repository. Design slides for this DAQ code are here, protocol spec is here, and the code is available here (under trunk/labview/labview-daq). It's stable and works well - we're using it on the compact RIO as well.
    2. The data on the web-based demo page is streamed from LabVIEW via this method.

    3. The NEES code uses ASCII, and so is limited in rate. One faster option is to use the Capture program by sending UDP from Labview. This works up to tens of kilohertz and has less overhead than TCP. Code is here as of 10/2008.
    4. LabVIEW has the DataSocket tools, which are very cool, but after discussions with NI we determined that we'd have to write our own non-trivial parser to convert the packets into DataTurbine datatypes, and at the moment the need just isn't there.
    5. Protocol buffers are another interesting idea, but at the moment there's no extant LabVIEW code to create them. Since LV can invoke C via DLLs, it's do-able but would need some sort of project requirement.
    6. There's the ActiveX bridge for windows, which I've not used.

    The above are all for streaming numeric data. It's worth noting that the NEES Labview code also handles control of hardware such as motors, via a separate protocol/daemon, so we have some expertise in this area.

    Making SAPI calls from Matlab

    The "Matlab" folder located in your DataTurbine installation contains a number of ".m" files which provide examples of calling the DataTurbine simple API (SAPI) from Matlab. SAPI calls are easily made from Matlab, due in no small part to Matlab's tight integration with Java.

    Before you can make SAPI calls from Matlab, you will need to add the "rbnb.jar" file to Matlab’s classpath. To do this, startup Matlab and then at the Matlab command prompt type "edit classpath.txt". Then add the full path to the "rbnb.jar" file to this file. As an example, here is a snippet from "classpath.txt":

    ##
    ## FILE: classpath.txt
    ##
    ## Generated from gen_classpath.pl
    ##
    ## Entries:
    ## o path_to_jarfile
    ## o [glnxa64,glnx86,sol2,sol64,unix,win32,win64,mac,maci,maci64]=path_to_jarfile
    ## o $matlabroot/path_to_jarfile
    ## o $jre_home/path_to_jarfile
    ##
    C:\PROGRA~1\RBNB\V3.2B5\bin\rbnb.jar
    $matlabroot/java/patch
    $matlabroot/java/jar/util.jar
    $matlabroot/java/jar/widgets.jar
    mac=$matlabroot/java/jarext/aquaDecorations.jar
    etc
    etc
    etc

    Notice that the full path to the desired DataTurbine "rbnb.jar" file
    ("C:\PROGRA~1\RBNB\V3.2B5\bin\rbnb.jar") has been added as the first non-comment line in the file.

    After editing "classpath.txt", save it, and then restart Matlab (so it will pick up the new classpath.txt file).

    To make sure everything is working, do the following after editing classpath.txt:

    1. In a command/terminal window, startup a DataTurbine server.
    2. Start Matlab
    3. In Matlab, cd to the "Matlab" folder located in your DataTurbine installation.
    4. From the Matlab command prompt, run "testrbnb.m". This executes the server
    test suite; various DataTurbine source/sink tests are executed. You will
    see lots of activity in the DataTurbine server window, and at the Matlab
    command prompt you will see lines like:

    >> testrbnb
    PASS: open/close test
    PASS: testput1
    PASS: testput2
    PASS: testput3
    PASS: testmulti
    PASS: multiple ring buffer test
    etc
    etc
    etc

    If you are able to run the "testrbnb.m" script as described above, then you are set to try out SAPI calls on your own from the Matlab command prompt. Look at the "testput1.m" file to start. Some convenience functions have been written, such as rbnb_source.m (creates a source and opens an RBNB connection) and rbnb_cmap.m (creates a channel map). You can look at the testput1.m file
    and try manually executing each command from the Matlab command prompt.

    Open Source DataTurbine on NMI System

    On October 1 2007, we completed our first successful build of Open Source DataTurbine software on NMI Build & Test facility. Since then, we are doing a nightly build of DataTurbine software on four different platforms. DataTurbine build runs overview can be found at:
    http://nmi-web.cs.wisc.edu/nmi/index.php?page=results/overview&rows=20&o...

    put/get units for channels

    Overview
        Many would like to annotate data channels with units metadata. Physical units are an example of DataTurbine's facility to associate any string with data channels.

    Procedure
        This is accomplished by using the function PutUserInfo() of the ChannelMap class in com.rbnb.sapi, the DataTurbine Simple API. The convention that is used is to prefix the unit label with "units="

    Examples
        For some channel 'foo', whose data has units 'bar' for which a DataTurbine Channel Map exists, do:
     PutUserInfo(GetIndex("foo"), "units=bar");
    in the Source program. This metadata can then be retrieved by Sink clients by:
    GetUserInfo(GetIndex("foo");
    after a ChannelMap has been populated with channel 'foo'.

        An example of a Source program that insrts unit metadata in this way is edu.sdsc.cleos.DaqToRbnb, which can be viewed here:
    http://nladr-cvs.sdsc.edu/svn-public/CLEOS/cleos-rbnb-apps/trunk/src/edu...
        Examples of Sink clients that use this unit metadata have complied to the protocol that prefixes unit metadata with 'units=' are plot.jar, which ships with DataTurbine and the Real-time Data Viewer from SUNY at Buffalo, which can be found here:
    http://it.nees.org/software/rdv/index.php

    Starting Mirrors from SAPI

    Beginning with DataTurbine V3.2B5, it is possible to start Mirrors using the "simple API" (SAPI) programming interface. These methods have been added to SAPI’s "Control" class. Most useful to the end user, 2 convenience functions have been added which allow the user to easily mirror a source starting "now" and going indefinitely. With the addition of these methods, the user now has 3 ways to start mirrors:

    1. Just as has always been done, continue to setup mirrors through rbnbAdmin.

    2. Use a simple Java program such as the following. This program will mirror a source named "rbnbSource" from the DataTurbine at "localhost:3333" to an output source named "mirrorSource" in the DataTurbine at "localhost:4444".


    import com.rbnb.sapi.Control;
    public class MirrorTest {
    public static void main(String[] argsI) throws Exception {
    Control cont = new Control();
    cont.OpenRBNBConnection("localhost:3333","tempConnection”);
    // Can specify cache/archive or to match the originating source using the following:
    // cont.CreateMirrorOut("rbnbSource","localhost:4444","mirrorSource",0,0,(byte)0,true);
    // To automatically match the originating source:
    cont.CreateMirrorOut("rbnbSource","localhost:4444","mirrorSource");
    cont.CloseRBNBConnection();
    }
    }

    3. Matlab can be used to setup the mirrors, as the following example shows. In this example, a source named "rbnbSource" from the DataTurbine at "localhost:3333" will be mirrored to an output source named "matlabMirrorSource" in the DataTurbine at "localhost:4444".


    >> c = com.rbnb.sapi.Control;
    >> c.OpenRBNBConnection('localhost:3333','tempConnection');
    >> c.CreateMirrorOut('rbnbSource','localhost:4444','matlabMirrorSource');
    >> c.CloseRBNBConnection;

    Time selection parameters in the Request SAPI call

    Terry Weymouth from UMich figured this out, this is a re-post of his findings.

    Definitions

    Request(channelMap, time, duration, mode)

    Where...
    recTime is the time stamp on record,
    lastTime is the greatest time ring buffer for a channel,
    firstTime is the smallest time in the ring buffer for a channel,
    maxLastTime is the greatest lastTime over all channels,
    minFirstTime is the smallest firstTime over all channels
    (for all duration >= 0 is required)

    absolute
    return records for which time <= recTime <= time+duration
    duration=0 is a special case, returns the single record with
    recTime <= time

    newest
    return records for which lastTime-(time+duration) <= recTime <=
    lastTime-time; duration = 0 is a special case, returns the single closest
    record to lastTime-time such that recTime <= lastTime-time

    oldest
    return records for which firstTime+time <= recTime <=
    firstTime+time+duration; duration = 0 is a special case, returns the single
    closest record to firstTime+time such that firstTime+time <= recTime

    aligned (e.g. newestAlligned)
    return records for which maxLastTime-(time+duration) <= recTime <=
    maxLastTime-time

    [did you-all ever consider oldestAlligned that is: minFirstTime+time <=
    recTime <= minFirstTime+time+duration???]

    after
    given baseTime is the max of lastTime-duration and time
    return records such that baseTime <= recTime <= baseTime+duration

    modified
    if time < lastTime, compute baseTime is lastTime-duration
    return records such that baseTime <= recTime <= baseTime+duration
    if time >= lastTime no data is returned

    next
    given that baseTime is the least recTime such that time <= baseTime
    return records for which baseTime <= recTime <= bastTime+duration

    previous
    given that baseTime is the greatest recTime such that baseTime <= time
    return records for which baseTime-duration <= recTime <= baseTime

    Anomalies

    'Next' with time < minFirstTime, duration > 0 

    and

    'Previous' with maxLastTime < time, duration >0

    both return a list an array of times/data with duplicates in each
    position but the first and the last
    'Previous' with time <= maxListTime and duration = 0

    only returned data on channel 0

    Comments and more information

    See the SAPI page on Request for more information.

    Usage of the DataTurbine KML plugin

    TrackDataPlugIn and TrackKMLPlugIn are DataTurbine PlugIns which format alt/lat/lon DataTurbine channel data into a KML document for viewing in Google Earth. These PlugIns come with the DataTurbine distribution; they are located within the Apache Tomcat directory at:

    apache-tomcat-6.0.18/webapps/webTurbine/WEB-INF/classes

    This document provides some information on using these PlugIns, as well as how to setup Google Earth for viewing the track data returned from these PlugIns.
     
    TrackDataPlugIn
    =============
    TrackDataPlugIn fetches the low-level data (altitude, longitude, latitude) from the DataTurbine channels and returns it to TrackKMLPlugIn (for final processing into a KML document).  As such, your DataTurbine source needs to contain alt, lat, and lon channels. The default channel names assumed by TrackDataPlugIn are "Alt", "Lat", and "Lon".  If your DataTurbine Source doesn't use these exact channel names, you will need to specify your channel names in a TrackDataPlugIn config file, and then on the command line to start TrackDataPlugIn you use the "-f" command line flag to specify this config file's name.  Here is an example of this file:

            #
            # This file provides TrackPlugIn the names of the
            # alt, lat, lon, id, type, and classification RBNB channel names.
            #
            # Blank lines or lines that start with '#' will be skipped
            #
            # The left column indicates the type of channel (see codes below).
            # The right column provides the specific RBNB channel name this data will be acquired from.
            #
            # The channel name codes are as follows (these names must be exactly used in the left column):
            # alt (altitude channel)
            # lat (latitude channel)
            # lon (longitude channel)
            # id (track id channel)
            # type (type channel)
            # class (classification channel)
           
            alt GPS_Alt_MSL
            lat Latitude
            lon Longitude

    When you launch TrackDataPlugIn, the following items must be included in the classpath: the path to the TrackDataPlugIn classes and also the path to rbnb.jar.  Here is an example:

            java -cp C:\PROGRA~1\RBNB\V3.2B5\apache-tomcat-6.0.18\webapps\webTurbine\WEB-INF\classes;C:\PROGRA~1\RBNB\V3.2B5\bin\rbnb.jar TrackDataPlugIn -f TrackDataConfig.txt -n TrackData

    The "-f" flag specifies the name of the config file and the "-n" flag specifies the name that will be registered for your PlugIn with the DataTurbine server (that is, the name that you would use for making data requests).  If you run TrackDataPlugIn with a "-h" flag you can check out the other command line options available.

    TrackKMLPlugIn
    =============
    TrackKMLPlugIn receives raw data (altitude, latitude, longitude) from TrackDataPlugIn and formats this data for output as a KML document.  This PlugIn uses an input config file to specify various options for how the track will look in Google Earth.  This file is specified with the "-f" command line option when launching TrackKMLPlugIn.  Here is a sample config file:
            #
            # This file specifies GUI options for TrackKMLPlugIn
            #
            # Blank lines or lines that start with '#' will be skipped
            #
           
            #
            # Display parameter options
            # Parameter Options
            # --------------------------------------------------------------------------------------------------
            # labelBy id/type/class/alt/latlon/altlatlon/none; any other entry will be treated as the constant label
            # sortBy id/type/class
            # iconBy type/class/dot/square/airplane/none
            # colorBy type/class/red/green/blue/yellow/purple/aqua; any other entry will be treated as the constant color (must be HEX number in format bbggrr)
            #
           
            labelBy Foo
            sortBy id
            iconBy airplane
            iconScale 1.0
            colorBy red
           
            #
            # To add current values as part of the label:
            # label_chan <chan1>=<nickname1>,<chan2>=<nickname2>,etc.
            #
            # - "chan" specifies a relative channel within the same Source as the track data is fetched from.
            # In other words, "chan" cannot be an absolute Source/Channel name.
            # - The "nickname" specifies the string that will be displayed in Google Earth with the data value.
            #
            label_chan GPS_Alt_MSL=Alt(m)
    In this sample config file, we specify to use an airplane icon in Google Earth, label the icon "Foo", and color the track red.  This file also specifies to display the Altitude value along with the aircraft icon in Google Earth.
    When you launch TrackKMLPlugIn, the following items must be included in the classpath: the path to the TrackKMLPlugIn classes and also the path to rbnb.jar.  Here is an example:

            java -cp C:\PROGRA~1\RBNB\V3.2B5\apache-tomcat-6.0.18\webapps\webTurbine\WEB-INF\classes;C:\PROGRA~1\RBNB\V3.2B5\bin\rbnb.jar TrackKMLPlugIn -C -f TrackKMLConfig.txt -n TrackKML
     
    The "-f" flag specifies the name of the config file and the "-n" flag specifies the name that will be registered for your PlugIn with the DataTurbine server (that is, the name that you would use for making data requests).  If you run TrackKMLPlugIn with a "-h" flag you can check out the other command line options available.

    Viewing tracks in Google Earth
    ========================
    The following software needs to be running for a complete end to end solution:

    • DataTurbine server
    • An instance of TrackDataPlugIn and TrackKMLPlugIn
    • Your DataTurbine Source application, which puts alt/lat/lon data into the DataTurbine
    • Apache Tomcat server (comes with the DataTurbine distribution); supports an HTTP interface for fetching data from the DataTurbine
    • Optional: TimeDrive server (comes with the DataTurbine distribution); provides a TiVO-like interface for requesting historical or real-time data from the DataTurbine server
    • Google Earth, to view the KML documents fetched from the DataTurbine

    Here are some instructions on getting the system going with Google Earth:

    1. Start DataTurbine server, Apache Tomcat server, your Source application, and instances of the track PlugIns
    2. Start Google Earth and create a "NetworkLink" object (which is used to periodically fetch KML documents via a URL).
      • The URL to include in the NetworkLink will be something like: http://localhost/RBNB/TrackKML/TrackData/Foo?d=500
      • Here, we are using "Foo" for the name of the DataTurbine source.  This URL requests 500 seconds worth of the most recent data (note the "?d=500" munge).
    3. If you want to create an even "slicker" system, instead of using a static URL such as the one above, start an instance of TimeDrive and allow the user to play/step through their data.
      • An instance of the TimeDrive server needs to be started; timedrive.jar is located in the same directory as rbnb.jar.
      • By default, TimeDrive uses port 4000.  If you need to specify a different port, you can use the "-s" command line flag.  If you use a different port, a further tweak needs to be made in the Tomcat "TimeDrive" servlet configuration file, located at "apache-tomcat-6.0.18\webapps\TimeDrive\WEB-INF\web.xml".
      • An example of launching TimeDrive: java -jar C:\PROGRA~1\RBNB\V3.2B5\bin\timedrive.jar -u localhost -d0 -m 2
      • When using TimeDrive, the URL you specify in the NetworkLink in Google Earth must change to something like: http://localhost/TimeDrive/RBNB/TrackKML/TrackData/Foo; note that we no longer need the time duration munge ("?d=500") because the user will specify start time and duration for the data request using the TimeDrive web interface.
      • To see the TimeDrive web interface, go to the following URL in a browser: http://localhost/TimeDrive
    4. You can also provide a Google Earth "setup" file for users, which specifies a desired collection of NetworkLinks.  The user simply loads this file into Google Earth (using File->Open).  Here is an example of a simple setup file.  In this case, Google Earth will automatically request a new KML document every 3 seconds.

      <?xml version="1.0" encoding="UTF-8"?>
      <kml xmlns="http://earth.google.com/kml/2.1">
      <Document>
      <Folder>
              <name>Foo</name>
              <description>
              <![CDATA[<a href="http://localhost/TimeDrive">TimeDrive</a>]]>
              </description>
              <open>1</open>
              <NetworkLink>
                      <name>Foo</name>
                      <visibility>0</visibility>
                      <open>0</open>
                      <Url>
                              <href>http://localhost/TimeDrive/RBNB/TrackKML/TrackData/Foo</href>
                             <refreshMode>onInterval</refreshMode>
                             <refreshInterval>3</refreshInterval>
                      </Url>
              </NetworkLink>
      </Folder>
      </Document>
      </kml>

    Using cat5 cable for serial cabling

    Many instruments communicate over serial links, such as RS-232. It is often the case that such instruments will serve as DataTurbine data sources. Wiring these links with ubiquitous standard cat5 or 6 networking cables proves to be a significant convenience in our experience. Simply create a DCE/DTE adapter pair with these DB9/RJ45 adapters: http://www.cyberguys.com/templates/SearchDetail.asp?productID=802 According to this standard: http://yost.com/computers/RJ45-serial/
    Here is a simple wiring guide to get it done:

    db9/rj45 Adapter Wiring

    DTE  DB9pin    DCE

    O           1          Br

    Bk          2          Y

    Y            3          Bk

    Br           4          O

    R,G         5         G,R

    -             6          -

    W            7         Bl

    Bl            8         W

    -              9         -

     
    Having done this, one can use standard networking cables to wire serial links.

    WebDAV walkthrough from Creare

    Original URL: http://rbnb.creare.com/rbnb/WP/RBNB_WebDAV_WalkThrough.html

    RBNB V2.0 WebDAV Interface

    Getting Started: A Walk Through

    March 3, 2003

    This is a walk-through to introduce new users to the RBNB WebDAV interface.  It starts from the point where you have downloaded and installed the RBNB WebTurbine package.  This is an evolving document that is expected to be further refined over time.

    1) Start WebTurbine

    Use the desktop icon or command-line per download instructions to startup the delivered Tomcat-based web server.

    2)  Visit the WebTurbine Home Page

    With a browser, go to URL localhost:8080.  You should see a WebTurbine welcome page.

    3) Start RBNB Server

    Click on the rbnbServer link, then click 'Start Server' using the default entries.  This should launch an RBNB server on your local machine. (It should be possible to start the RBNB outside of Tomcat and have it work, but this way ensures Tomcat is running and makes launching RBNB easier for first time users).

    4) Review RBNB Content with Browser

    With your browser, go to URL localhost:8080/RBNB.  You should see a "folder" entry of "Server".  Click it.  You are now looking at the local RBNB sources, which by default should include "_Metrics" and "_Log".  Go in the _Log directory, and click one or more items to see if you can retrieve the text log messages.

    5) Mount RBNB as WebDAV File System

    Now lets try to mount the RBNB/WebDav server as a file system.  In all cases, the <RBNB-URL> will be of the form "http://your-machine:8080/RBNB" ("localhost" may or may not work).

    On a PC, do one of the following:
        a) From Internet Explorer, choose File/Open...  Click "Open as Web Folder". Enter the <RBNB-URL>
        b) From the Windows File Explorer (on XP), select "Map Network Drive" and enter the <RBNB-URL>

    On a Mac, do one of the following:
        a) From the Go/Connect to Server pull-down and type in the <RBNB-URL>
        b) From a command line, using mount_webdav:
            mount_webdav <RBNB-URL> /Volumes/RBNB (or whatever mount point you want)

    Confirm the file system is mounted using standard File Explorer or other interface.

    Note:
    On PCs, the IE "Web Folder" is a GUI that lets you drag and drop content, but does not mount a true file-system interface available to other applications.  The "Map Network Drive" option is only available on XP, and must use the standard web server port 80 (not 8080).  Edit the Tomcat server.xml file to change this. Third party WebDAV filesystem clients for various Windows OS are also available.

    Warning:
    Under some circumstances killing Tomcat while an RBNB WebDAV is mounted leaves an orphaned filesystem that is difficult to unmount.

    6) Create RBNB Source and Channel via WebDAV

    From a command line shell, go to the mounted filesystem, create a folder (new source), and create a data channel. (Here, Unix syntax is used, but equivalent PC/DOS commands will work).

    $ cd /Volumes/RBNB/Server
    $ mkdir foo
    $ cd foo
    $ echo hello > c@t=1
    $ cat c
    hello
    $ echo world > c@t=2
    $ cat c@t=1
    hello
    $ cat c@t=2
    world
    $ cat "c@t=1&d=2"
    hello
    world
    $

    Notes:
    The @timestamp "munge" has 3 main keys, separated by the '&' symbol, e.g:

    chan@t=1&d=2&r=newest

    t: timestamp (seconds)
    d: duration (seconds)
    r: reference (oldest | newest | absolute)

    The default with no-munge is simply "t=0&d=0&newest".  If a time is specified, the default reference is "absolute". Absolute time is seconds since midnight Jan 1 1970 GMT. On write, "newest" means current time of day.

    Quotes around the filename-with-munge are not always necessary, but depending on the OS may be required because of the treatment of the special '@' and '&' characters.

    Warning:
    With RBNB V2.0, all channels put to RBNB must have increasing timestamps.  Thus, you can't put in a@t=2 followed by b@t=1.  This changes in a future release, where we use independent RBOs for every channel (vs one for each source).  In many cases it is best to simply use automatic time-of-day timestamps for data sources.

    Warning:
    Most file systems and web browsers try really hard to cache content.  This means after you ask for the default "newest" data from file "x" once, you may keep getting the same (cached) value even if newer data is available.  One simple work around for this is to use an (otherwise meaningless) increment, as in:

    $ echo 1 > x
    $ cat x
    1
    $ echo 2 > x
    $ cat x
    1
    $ cat x@i=1
    2
    $ echo 3 > x
    $ cat x@i=2
    3
    $ cat x@d=99
    1
    2
    3

    7) User Other Applications to Create and View Content

    After creating some channels via the file system interface, try reviewing them with other applications:

    Open an rbnb file (channel) using Notepad or other text editor.  Save the result.  Review the earlier and most recent versions via the file system:

    $ cat myfile.txt
    $ cat myfile.txt@r=oldest

    Use rbnbChat to send messages, then look at the messages via the file system or browser.

    Warning/Notes:
    The RBNB WebDav interface does not yet fully support file copying/renaming/deleting.  Some applications do a save-to-temporary file then move it to the real name.  This may fail and/or strand RBNB channels of the temporary names.

    8) Mount the RBNB WebDAV as File System on Other Machines

    Mount the RBNB as file system from other machines:

    • Web Folder, part of MS Internet Explorer
    • "Map Network Drive" via Windows XP File Explorer
    • davfs for Linux (standard with Mandrake Linux, available for others)
    • WebDrive for windows from South River Technologies
    • WebFile Client from Xythos

    Warning/Notes:
    Each of these other WebDav-as-Filesystem clients has its own quirks.  For example, the XP map network drive will not recognize a port, so you have to run your Tomcat at port  80.  To do this, edit the jakarta/config/server.xml file, and (if Unix) run the tomcat server as root.

    9) Publish Conventional Files via rbnbFolder

    Use the rbnbFolder PlugIn (available as a link on the RBNB Welcome web page) to point to a traditional file folder on your hard drive.  See these reflected as RBNB channels via the WebDAV interface.  Note how this lets you intermix "real" files with RBNB content as seen by web browser and other clients.

    10) Route RBNB Servers Together

    Connect RBNB servers in parent/child routed connections.  Peruse content from remote servers via the WebDAV interface to your local machine.  Try using the RBNB PlugIn rbnbFolder to "publish" remote (real) file content.

    Note:
    You can only create new folders on the server to which you are directly connected.  This is because your WebDAV interface is the "source" for locally created content, and your local WebDAV server does not have direct rbnb source connections to the remote servers.  Put another way, you have write privilege locally and read-only privilege to remote/routed servers.

    11) Create Virtual Data Processing Channels

    This exercise sets up a more sophisticated scenario similar to the on-line demo at http://rbnb.creare.com/rbnb/rbnbWeather.html.

    a) Start an RBNB server using default settings
    c) Start the PNGPlugIn using default settings
    d) Start rbnbSource, connect to RBNB, click Start
    e) Start a Virtual Source Plugin with Origin: "PNGPlugIn/rbnbSource"
    d) Use your web browser to look at the resulting PNG plot, e.g:

    http://my-machine:8080/RBNB/Server/VSource/c0@d=1

    e) Review the resulting files and folders via the WebDAV file interface.

    Note that this demo creates a streaming data source that you can repeatedly read via any application that can be set to continuously open/read the associated URL or WebDAV file.

    WebDAV: How to set cache and archive values when creating a source directory

    From the RBNB-dev mailing list, how to set the cache and archive parameters.

    As background, creating a directory in an WebDAV-mounted filesystem is the same as creating a new source in RBNB. If you use special syntax, you can tell DataTurbine how much memory and disk to set aside for this new source. Here, cribbed from the email discussion, are the instructions:

      mkdir foo@"a=100000&c=1000"
    

    Where a is archive and c is cache, in either frames or seconds. See the TrimByTime FAQ entry for more details.