Hi,

Here is the summary of the activities from June 27th to July 10th 2009
(for two weeks).

NTL update:
The overall plan for the summer is to deploy on-ramp code, and then
the off-ramp code. The onramp part is deployed and has been up. As a
next step, we will deploy the off-ramp code. By the end of July, we
should have a working deployment.

Please use the RDV to view the latest data:
Address = 198.150.174.55
Port number = 3333

MCR update:
Tony submitted the paper to the ISSNIP 2009. Thank you for leading
the effort. He returns next week from his field research on the coral
reefs around Saba and St. Martin.

Several components of the MCR system were upgraded last week. Since
then, it has been up. Here are the details of the upgrade:

- 220V UPS is installed
- 220V DSL modem replaced the older Peabird modem, and it is properly
configured.
- Seacat 16 plus was taken out and put back. Connecting the cable to
the power line stopped sending the data. Right now, it is operating
on a battery.
- Axis camera program is re-written for IP changes and network failures.
- igaussu.sdsc.edu has a new version of Linux kernel (OS upgrade).

We also added event detection channels. Please check out the event
results from the data server:
host name: iguassu.sdsc.edu
port number: 3333

The streaming data is being constantly monitored for low salinity and
high rain events. We will work with Libe to tune the parameters. As
a next step, we will work on bleaching conditions with Andy.

Just as a note, we received the UPS for the downlink node. However,
the new downlink node (Dell), which was ordered more than three weeks
ago, is still not here. Dell is lagging, and it will be here in a
couple of weeks according to the schedule from its website.

Meeting reminder:
- Right after the ISSNIP 2009, Scott is proposing CREON meeting
(December 10th and 11th).
- Please mark your calendar for Coral Reef Workshop (March 5,
2010) at UCSD next year.

0. Write the start-up script in case of reboot or crash:
status: in progress, implementing

1. Write up the instruction for using RDV for MCR streams.
status: in progress

2. Refine, harden, and document all of the code, and publish them online.
Status: The IP camera Axis source is changed for network failures.
Plan: The DT2DB is being modified to be more robust and simple. We
will deploy it next week.

3. ISSNIP paper
Status: submitted by Tony.
Deadline: June 15
ISSNIP Website: http://www.issnip.org/2009/
ISSNIP CFP: http://www.issnip.org/2009/ISSNIP2009CFP2.pdf

4. Integration of DT with Event detection using complex event processing engine
status: it is deployed in iguassu.sdsc.edu
plan: we will explore different types of queries and start measuring
the performance.

5. Wireless options for buoy
Status: surveyed WiMax
Plan: write up the summary after the paper deadline.

6. DB for DT
status: the free, open-source DB postgresql is deployed, and the DB is
created.
plan: write a script to create tables, configure DT2DB, and deploy it next week.

MCR call reminder:
We will have a regular weekly call on Wednesday July 15, 2009 from 11
a.m. to 12 p.m.

Thank you,

Sameer and Peter

Hi,

Here is the summary of the activities from June 20th to June 26th 2009.

NTL testbed update:
Luke restarted the DT server in NTL.
Host: 198.150.174.55
Port number: 3333

UCSD has contacted Lauri Arvola to understand details of his new
deployment. Lauri wants to discuss how it can be integrated with the
GLEON system.

UCSD will participate in the GLEON IT meeting in the 1st week of July.

MCR update:
This week, we focused our efforts on various aspects of the ISSNIP paper.

The linux kernel in iguassu.sdsc.edu is upgraded in order to
accommodate cloud computing platforms. We will configure the DT
servers in them next week.

Meeting reminders:
Right after the ISSNIP 2009, Scott is proposing CREON meeting
(December 10th and 11th).
Also, please mark your calendar for Coral Reef Workshop (March 5,
2010) at UCSD next year.

Vacation reminders:
UCSD:
During the first two weeks of July, Tony F. will be out of office for vacation.
From July 23 to July 27, Peter S. will take a vacation.

0. Write the start-up script in case of reboot or crash:
status: in progress

1. Write up the instruction for using RDV for MCR streams.
status: in progress

2. Refine, harden, and document all of the code, and publish them online.
Status: on going

3. ISSNIP paper
Status: the first draft is being circulated.
Deadline: June 30
ISSNIP Website: http://www.issnip.org/2009/
ISSNIP CFP: http://www.issnip.org/2009/ISSNIP2009CFP2.pdf

4. Integration of ESB-ESPER-OSDT
status: we are designing the overall architecture while some parts are
implemented.

5. Wireless options for buoy
status: in progress
Plan: write up the summary after the paper deadline.

6. DB for DT
status: in progress
plan: the new cloud computing platform has the PostgreSQL installed.
A new DB will be created, and a DT2DB will be deployed in next two
weeks.

7. cloud computing machine setup
status: completed

No MCR calls until the third week of July.

Thank you,

Sameer and Peter

Hi,

Here is the summary of the activities from June 13th to June 19th 2009.

NTL testbed update:
NTL will testbed activity will resume in early July.
So far, we set up the on-ramp machine, and it's been running fine for a week.
You can view the data using the RDV:
host: 198.150.174.55
port: 3333

MCR update:
Over the Wednesday phone call, we discussed the next phase deployment.

A former Seabird engineer, Andrew Zigwid (?) joined McCartney group,
and Keith is negotiating with him to put together a complete
deployment package, that includes ADCP, thermistor chain, inductive
modem, buoy, cabling and etc. The package will be ready to deploy in
the ocean. Libe liked the idea, and raised the concern about the data
rate of ADCP through inductive modem. Andy will follow up on the
question. Keith is the main point of contact for this activity to the
McCartney group.

Regarding a buoy, Kim Page-Albins is the person Peter S. contacted.
She did not have the ADCP as part of her deployment for PISCO.
http://osu.piscoweb.org/data/sbe162823.html

However, Kim told me that Oregon state does have an ADCP in one of the buoy.
http://agate.coas.oregonstate.edu/data/nh10.html

Next week, Andy will set up a Skype number so that UCSD can call to
Moorea over a phone line. With Sally, Russ and everyone else, we will
discuss various plans and coordinate our activities.

Last weekend, data acquisition box had an unexpected error in one of
the previously installed software (Norton Ghost). The error affected
network connectivity of the windows machine. Give that a new machine
is ordered, and that this error will not likely to recur, we will not
try to fix it fundamentally - we just restarted the drivers, and it's
working.

Meeting reminders:
Right after the ISSNIP 2009, Scott is proposing CREON meeting
(December 10th and 11th).
Also, please mark your calendar for Coral Reef Workshop (March 5,
2010) at UCSD next year.

This week, we focused on writing the paper as well as integrating the
DataTurbine with complex event processing engine (ESPER).

Vacation reminders:
UCSD:
During the first two weeks of July, Tony F. will be out of office for vacation.
From July 23 to July 27, Peter S. will take a vacation.

0. Write the start-up script in case of reboot or crash:
status: in progress

1. Write up the instruction for using RDV for MCR streams.
status: in progress

2. Refine, harden, and document all of the code, and publish them online.
Status: DT sink proxy for Jython is implemented. ESPER integration
pieces are being published. The doc and the code are here:
http://code.google.com/p/oss-dataturbine/source/browse/#svn/trunk/apps/o...

3. ISSNIP paper
Status: in progress, we have been discussing and writing various
parts. Thanks everyone who has been helping.
Deadline: June 30
ISSNIP Website: http://www.issnip.org/2009/
ISSNIP CFP: http://www.issnip.org/2009/ISSNIP2009CFP2.pdf

4. Integration of DT with Event detection using complex event processing engine
status: we implemented integration piece, and we are improving it to
be more efficient as well as more generally applicable.
plan: we will conduct various performance tests, and we will write
design and user manuals.

5. Wireless options for buoy
Status: surveyed WiMax
Plan: write up the summary after the paper deadline.

6. DB for DT
status: in progress
plan: configure the postgresql DB, and deploy DT2DB within two weeks.
This will run on cloud computing machine, and we are waiting it to be
available through an external IP.

7. cloud computing machine setup
status: a cloud computing platform Xen is installed in
iguassu.sdsc.edu, but the external IPs are not set up correctly. UCSD
sysadmin are working on it.

MCR call reminder:
Andy suggested having the call at a regular weekly call on Wednesday
June 24, 2009 from 11
a.m. to 12 p.m.

Thank you,

Sameer and Peter

Hi,

Here is the summary of the activities from June 27th to July 10th 2009
(for two weeks).

NTL update:
The overall plan for the summer is to deploy on-ramp code, and then
the off-ramp code. The onramp part is deployed and has been up. As a
next step, we will deploy the off-ramp code. By the end of July, we
should have a working deployment.

Please use the RDV to view the latest data:
Address = 198.150.174.55
Port number = 3333

MCR update:
Tony submitted the paper to the ISSNIP 2009. Thank you for leading
the effort. He returns next week from his field research on the coral
reefs around Saba and St. Martin.

Several components of the MCR system were upgraded last week. Since
then, it has been up. Here are the details of the upgrade:

- 220V UPS is installed
- 220V DSL modem replaced the older Peabird modem, and it is properly
configured.
- Seacat 16 plus was taken out and put back. Connecting the cable to
the power line stopped sending the data. Right now, it is operating
on a battery.
- Axis camera program is re-written for IP changes and network failures.
- igaussu.sdsc.edu has a new version of Linux kernel (OS upgrade).

We also added event detection channels. Please check out the event
results from the data server:
host name: iguassu.sdsc.edu
port number: 3333

The streaming data is being constantly monitored for low salinity and
high rain events. We will work with Libe to tune the parameters. As
a next step, we will work on bleaching conditions with Andy.

Just as a note, we received the UPS for the downlink node. However,
the new downlink node (Dell), which was ordered more than three weeks
ago, is still not here. Dell is lagging, and it will be here in a
couple of weeks according to the schedule from its website.

Meeting reminder:
- Right after the ISSNIP 2009, Scott is proposing CREON meeting
(December 10th and 11th).
- Please mark your calendar for Coral Reef Workshop (March 5,
2010) at UCSD next year.

0. Write the start-up script in case of reboot or crash:
status: in progress, implementing

1. Write up the instruction for using RDV for MCR streams.
status: in progress

2. Refine, harden, and document all of the code, and publish them online.
Status: The IP camera Axis source is changed for network failures.
Plan: The DT2DB is being modified to be more robust and simple. We
will deploy it next week.

3. ISSNIP paper
Status: submitted by Tony.
Deadline: June 15
ISSNIP Website: http://www.issnip.org/2009/
ISSNIP CFP: http://www.issnip.org/2009/ISSNIP2009CFP2.pdf

4. Integration of DT with Event detection using complex event processing engine
status: it is deployed in iguassu.sdsc.edu
plan: we will explore different types of queries and start measuring
the performance.

5. Wireless options for buoy
Status: surveyed WiMax
Plan: write up the summary after the paper deadline.

6. DB for DT
status: the free, open-source DB postgresql is deployed, and the DB is
created.
plan: write a script to create tables, configure DT2DB, and deploy it next week.

MCR call reminder:
We will have a regular weekly call on Wednesday July 15, 2009 from 11
a.m. to 12 p.m.

Thank you,

Sameer and Peter

Hi,

Here is the summary of the activities from June 20th to June 26th 2009.

NTL testbed update:
Luke restarted the DT server in NTL.
Host: 198.150.174.55
Port number: 3333

UCSD has contacted Lauri Arvola to understand details of his new
deployment. Lauri wants to discuss how it can be integrated with the
GLEON system.

UCSD will participate in the GLEON IT meeting in the 1st week of July.

MCR update:
This week, we focused our efforts on various aspects of the ISSNIP paper.

The linux kernel in iguassu.sdsc.edu is upgraded in order to
accommodate cloud computing platforms. We will configure the DT
servers in them next week.

Meeting reminders:
Right after the ISSNIP 2009, Scott is proposing CREON meeting
(December 10th and 11th).
Also, please mark your calendar for Coral Reef Workshop (March 5,
2010) at UCSD next year.

Vacation reminders:
UCSD:
During the first two weeks of July, Tony F. will be out of office for vacation.
From July 23 to July 27, Peter S. will take a vacation.

0. Write the start-up script in case of reboot or crash:
status: in progress

1. Write up the instruction for using RDV for MCR streams.
status: in progress

2. Refine, harden, and document all of the code, and publish them online.
Status: on going

3. ISSNIP paper
Status: the first draft is being circulated.
Deadline: June 30
ISSNIP Website: http://www.issnip.org/2009/
ISSNIP CFP: http://www.issnip.org/2009/ISSNIP2009CFP2.pdf

4. Integration of ESB-ESPER-OSDT
status: we are designing the overall architecture while some parts are
implemented.

5. Wireless options for buoy
status: in progress
Plan: write up the summary after the paper deadline.

6. DB for DT
status: in progress
plan: the new cloud computing platform has the PostgreSQL installed.
A new DB will be created, and a DT2DB will be deployed in next two
weeks.

7. cloud computing machine setup
status: completed

No MCR calls until the third week of July.

Thank you,

Sameer and Peter

Hi,

Here is the summary of the activities from June 13th to June 19th 2009.

NTL testbed update:
NTL will testbed activity will resume in early July.
So far, we set up the on-ramp machine, and it's been running fine for a week.
You can view the data using the RDV:
host: 198.150.174.55
port: 3333

MCR update:
Over the Wednesday phone call, we discussed the next phase deployment.

A former Seabird engineer, Andrew Zigwid (?) joined McCartney group,
and Keith is negotiating with him to put together a complete
deployment package, that includes ADCP, thermistor chain, inductive
modem, buoy, cabling and etc. The package will be ready to deploy in
the ocean. Libe liked the idea, and raised the concern about the data
rate of ADCP through inductive modem. Andy will follow up on the
question. Keith is the main point of contact for this activity to the
McCartney group.

Regarding a buoy, Kim Page-Albins is the person Peter S. contacted.
She did not have the ADCP as part of her deployment for PISCO.
http://osu.piscoweb.org/data/sbe162823.html

However, Kim told me that Oregon state does have an ADCP in one of the buoy.
http://agate.coas.oregonstate.edu/data/nh10.html

Next week, Andy will set up a Skype number so that UCSD can call to
Moorea over a phone line. With Sally, Russ and everyone else, we will
discuss various plans and coordinate our activities.

Last weekend, data acquisition box had an unexpected error in one of
the previously installed software (Norton Ghost). The error affected
network connectivity of the windows machine. Give that a new machine
is ordered, and that this error will not likely to recur, we will not
try to fix it fundamentally - we just restarted the drivers, and it's
working.

Meeting reminders:
Right after the ISSNIP 2009, Scott is proposing CREON meeting
(December 10th and 11th).
Also, please mark your calendar for Coral Reef Workshop (March 5,
2010) at UCSD next year.

This week, we focused on writing the paper as well as integrating the
DataTurbine with complex event processing engine (ESPER).

Vacation reminders:
UCSD:
During the first two weeks of July, Tony F. will be out of office for vacation.
From July 23 to July 27, Peter S. will take a vacation.

0. Write the start-up script in case of reboot or crash:
status: in progress

1. Write up the instruction for using RDV for MCR streams.
status: in progress

2. Refine, harden, and document all of the code, and publish them online.
Status: DT sink proxy for Jython is implemented. ESPER integration
pieces are being published. The doc and the code are here:
http://code.google.com/p/oss-dataturbine/source/browse/#svn/trunk/apps/o...

3. ISSNIP paper
Status: in progress, we have been discussing and writing various
parts. Thanks everyone who has been helping.
Deadline: June 30
ISSNIP Website: http://www.issnip.org/2009/
ISSNIP CFP: http://www.issnip.org/2009/ISSNIP2009CFP2.pdf

4. Integration of DT with Event detection using complex event processing engine
status: we implemented integration piece, and we are improving it to
be more efficient as well as more generally applicable.
plan: we will conduct various performance tests, and we will write
design and user manuals.

5. Wireless options for buoy
Status: surveyed WiMax
Plan: write up the summary after the paper deadline.

6. DB for DT
status: in progress
plan: configure the postgresql DB, and deploy DT2DB within two weeks.
This will run on cloud computing machine, and we are waiting it to be
available through an external IP.

7. cloud computing machine setup
status: a cloud computing platform Xen is installed in
iguassu.sdsc.edu, but the external IPs are not set up correctly. UCSD
sysadmin are working on it.

MCR call reminder:
Andy suggested having the call at a regular weekly call on Wednesday
June 24, 2009 from 11
a.m. to 12 p.m.

Thank you,

Sameer and Peter

Hi,

Here is the summary of the activities from June 6th to June 12th 2009.

Today, our NSF manager visited us. We explained that various
communities use the DT, and displayed the data from several
communities using RDV. It went successfully. Thank you very much for
all the help.

This week, all the deployed sensors from both MCR and NTL sites
streamed in their data.

NTL update:
Luke, Sameer and Peter S., deployed the onramp deployment, and the
data from it was shown to the NSF manager. Many thanks to Luke for
setting this up in short time.

Plan:
Now from the deployed the on-ramp piece, we will configure and deploy
Vega DB with off ramp code and test by end of July.

MCR update:
Over this week's phone call, Andy and Tony discussed the next phase
deployment plan. We reset the next deployment date for MCR to be
around March and April next year. Gastil wanted to learn more about
the issues related to data models for MCR - Tony gave several pointers
to papers describing the standards from various related communities
(e.g. ocean, water and lake).

Data acquisition box rebooted today with Windows XP patching. We
restarted the DT server and other programs. We have been writing a
script to automate this process (0 in our task list)

Meeting reminders:
Right after the ISSNIP 2009, Scott is proposing CREON meeting
(December 10th and 11th).
Also, please mark your calendar for Coral Reef Workshop (March 5,
2010) at UCSD next year.

This week, we got all the channels back up for MCR, and discussed
various ideas for the papers.

0. Write the start-up script in case of reboot or crash:
status: in progress, implementing

1. Write up the instruction for using RDV for MCR streams.
status: in progress

2. Refine, harden, and document all of the code, and publish them online.
Status: in progress, xml configuration file is implemented, but needs
integrated with other programs.
Plan: In progress, LoggerNetSrc was deployed in NTL testbed.

3. ISSNIP paper
Status: in progress, we have been discussing and writing various
parts. Thanks everyone who has been helping.
Deadline: June 15
ISSNIP Website: http://www.issnip.org/2009/
ISSNIP CFP: http://www.issnip.org/2009/ISSNIP2009CFP2.pdf

4. Integration of DT with Event detection using complex event processing engine
status: we implemented integration piece, and we are improving it to
be more efficient as well as more generally applicable.
plan: we will conduct various performance tests, and we will write
design and user manuals.

5. Wireless options for buoy
Status: surveyed WiMax
Plan: write up the summary after the paper deadline.

6. DB for DT
status: in progress
plan: configure the postgresql DB, and deploy DT2DB within two weeks.

MCR call reminder:
We will have a regular weekly call on Wednesday June 17, 2009 from 11
a.m. to 12 p.m.

Thank you,

Sameer and Peter

Hi,

Here is the summary of the activities from May 30th to June 5th 2009.

NTL update:
Luke, Sameer and Peter S., had a brief call to plan the onramp
deployment next week.

Plan:
Next Tuesday at 2:30 p.m. – Luke and Peter S. will configure and
deploy the on-ramp code.
From mid-June to early July – Both Luke and Peter S. will be out of office.
By end of July – configure and deploy Vega with off ramp code and test.

MCR update:

Peabird modem is configured correctly with the Linksys router. CTD
and weather station are streaming the data. Camera needs a bit more
time, and Vinny will help one more time tomorrow before he leaves on
Sunday.

Weather station is up to date now. It was updated from the internal
memory of data logger.

However, saved historical data from MOXA boxes for CTD came in very
bad shape - roughly 1 out of 5 data points was not in the correct
format. Some of them had right format, but the values were incorrect.
Incorrect values are harder to detect in the current setup - this is
where QA/QC will shine.

Among the incorrect values, some data had future year values. Since
DT expects monotonically increasing time values, the CTD source will
discard the incoming data. So, we will capture the incoming data
through a new DT source while keeping the old one alive. We were
aware of these issues from working in the USArray project. We are
working towards the QA/QC in various perspectives already. This
situation gives good motivation to do what we are working on.

Next Friday, our NSF program manager will visit. We will demonstrate
various projects using DT.

Meeting reminders:
Right after the ISSNIP 2009, Scott is proposing CREON meeting
(December 10th and 11th).
Also, please mark your calendar for Coral Reef Workshop (March 5,
2010) at UCSD next year.

This week, we focused more on the paper and the event detection part.

0. Write the start-up script in case of reboot or crash:
status: in progress, implementing

1. Write up the instruction for using RDV for MCR streams.
status: in progress

2. Refine, harden, and document all of the code, and publish them online.
Status: in progress
Plan: In progress, LoggerNetSrc was deployed tested on another setup
(weather station in UCSD). Configuration file reader is being
designed.

3. ISSNIP paper
Status: in progress
Deadline: June 15
ISSNIP Website: http://www.issnip.org/2009/
ISSNIP CFP: http://www.issnip.org/2009/ISSNIP2009CFP2.pdf

4. Event detection using complex event processing engine
status: implemented, and being documented
We implemented the first low salinity check, and ran it through the
historical data. Since it rained spotty, we did not find an event
where it lasted for one hour. We will iterate on this as we move
forward.

5. Wireless options for buoy
Status: surveyed WiMax
Plan: write up the summary after the paper deadline.

6. DB for DT
status: in progress
plan: configure the postgresql DB, and deploy DT2DB early next week.

MCR call reminder:
We will have a regular weekly call on Wednesday June 10, 2009 from 11
a.m. to 12 p.m. Andy, will you be back? If not, we might cancel it
since Libe will be out of town.

Thank you,

Sameer and Peter

Hi,

Here is the summary of the activities from May 23rd to May 29th 2009.

NTL update:
Luke, Sameer and Peter S. had a teleconference to plan the action
items and deadlines. Luke acquired the machine this week.

Plan:
Next week – set up the machine
By mid-June – set up the on-ramp part, test and deploy
From mid-June to early July – Both Luke and Peter S. will be out of office.
By end of July – configure and deploy Vega with off ramp code and test.

MCR update:

The phone company came and fixed the line. We replaced the modem with
the old Peabird modem, and reset it. The modem is connecting to the
Internet now. However, the DynDNS or a router to the modem is not set
up with it.

The process is rather slow because there is no correct Peabird
configuration document, and everything is over the phone with Vinny,
who has been helping us a bunch.

Please mark your calendar for Coral Reef Workshop (March 5, 2010) at UCSD.

0. Write the start-up script in case of reboot or crash:
status: In progress implementing
plan: test it using virtual machines.

1. Write up the instruction for using RDV for MCR streams.
status: delayed, in progress (screen shots are taken)
plan: finish it up next week

2. Write up how to display a graph on a web page, from the
real-time streaming data using DT.
Status: attached

3. Refine, harden, and document all of the code, and publish them online.
Status: in progress
Plan: In progress, LoggerNetSrc was deployed tested on another setup
(weather station in UCSD). Configuration file reader is being
designed.

4. ISSNIP paper
Status: in progress
Deadline: June 15
ISSNIP Website: http://www.issnip.org/2009/
ISSNIP CFP: http://www.issnip.org/2009/ISSNIP2009CFP2.pdf

5. Event detection using complex event processing engine
status: in progress, on going.
We are implementing the queries, and we will test and deploy them next week.

6. Wireless options for buoy
Status: on-going.

MCR call reminder:
We will have a regular weekly call on Wednesday June 3, 2009 from 11
a.m. to 12 p.m.

Thank you,

Sameer and Peter

Hi,

Here is the summary of the activities from May 16th to May 22nd 2009.

NTL update:

We worked with Chris Solomon to synthesize his notes on QA in the
GLEON data analysis projects. We plan to continue this activity and
to generate a document that captures his experiences and provides
recommendations on QA approaches and infrastructure.

Status on the testbed:
- A new Machine in NTL is ordered, and no ETA yet.
- Luke will wait for the new machine to arrive before setting up the
DT, and in the meantime, he reviewed both on-ramp and off-ramp codes.
- On ramp version is changing. Peter S. will send a new version of
LoggerNetSrc code to Luke when it gets completed (next week).

MCR update:

The connection in Moorea is still down. The people in the MCR station
contacted the phone company, and we are waiting on them.

We discussed the project that shares the infrastructure goals:
http://www.openioos.org.

The architecture and various perspectives on coral reef monitoring
will be discussed at the workshop next year. Please mark your
calendar for Coral Reef Workshop (March 5, 2010) at UCSD.

Here is the progress update on the action items.
0. New data acquisition box.
Status: new
Detail: The current machine is four years old, and has various other
software components installed. While the streaming data is down, we
will replace it with a new machine for the reliability of the system.
We are in the process of ordering a machine.

1. Write the start-up script in case of reboot or crash:
status: Started implementing it, and needs testing
plan: test it using virtual machines.

2. Write up the instruction for using RDV for MCR streams.
status: delayed, in progress (screen shots are taken)
plan: finish it up next week

3. Write up how to display a graph on a web page, from the
real-time streaming data using DT.
Status: delayed, in progress.

4. Refine, harden, and document all of the code, and publish them online.
Status: in progress
Plan: Next week, Peter S. will separate out the configuration
parameters into a file, and after testing them, the code for LoggerNet
will be sent to NTL.

5. ISSNIP paper
Status: in progress
Deadline: June 15
ISSNIP Website: http://www.issnip.org/2009/
ISSNIP CFP: http://www.issnip.org/2009/ISSNIP2009CFP2.pdf

6. Event detection using complex event processing engine
status: in progress, on going.
We are implementing the integration of streaming data middleware and
the complex event processing engine while translating the queries from
UCSB into a ESPER query language.

7. Wireless options for buoy
Status: on-going. We are looking into both 802.11n and WiMax. We
will survey further and report the findings. The focus of the survey
includes the following parameters:
- reliability during the rainy weather.
- range
- the price
- power consumption
Plan: Survey further, and then contact the others with experiences.
Write up the findings (probably after ISSNIP deadline).

MCR call reminder:
We will have a regular weekly call on Wednesday May 27, 2009 from 11
a.m. to 12 p.m.

Thank you,

Sameer and Peter

Hi,

Here is the summary of the activities from May 9th to May 15th 2009.

NTL update:

NTL is busy getting ready for summer deployment for next few weeks.
Next few weeks, we will hold off the regular Friday calls while UCSD
and NTL will set up the hardware and the software pieces in each site
for the testbed.

Chris Solomon and Tony Fountain discussed QA/QC scenarios this week,
and Chris sent UCSD the case studies. Next week, UCSD will go over
them and see how they fit within the streaming data middleware and the
complex event processing framework.

MCR update:

Last week, we determined that we had a bad router. This week, Vinny
diagnosed the problem further, and found out that the phone line also
had a problem (there is no tone when a phone is connected). In the
MCR station, it is possible that the phone lines get cut or get
corroded.

Prior to finding out that the phone line was bad, we tried connecting
to the Internet with a new router too. There are three DSL modems in
the lab, not counting the recently broken one. We set up one of them,
but while the router’s web interface worked, the router never
connected to the Internet.

Andy contacted the MCR station regarding this phone line problem. For
the router, instead of using the newly setup DSL modem, we will use
the P-Bird modem that has worked reliably in the past, or purchase a
new one.

Someone from UCSB is planning to visit Moorea on the June 3rd. UCSD
is currently researching the best router. The long term solution of
220V UPS will have to wait because heavy items are shipped together
through a mail (UPS along with the engine – for a boat?).

Coral Reef Workshop (May 5 or 12, 2009):
Please check your schedule and tell us which date works better for
you: May 5 or 12, 2010. Please also tell us if any of the dates do
not work for you.

Here is the progress update on the action items.
0. While the streaming data is down, we will either purchase a new
machine or put a fresh OS in the data acquisition machine to increase
the reliability of the system.
Status: new

1. Survey a good router and recommend it early next week.
Status: new

2. Write the start-up script in case of reboot or crash:
status: started, in progress
plan: We want to automate the recovery process from the power failure.
We will write this script as generic as possible so that it can be
ported to other machines.

3. Write up the instruction for using RDV for MCR streams.
status: in progress
plan: write it up next week.

4. Create an example of how to display a graph on a web page,
from the real-time streaming data using DT.
Status: in progress
Plan: write it up next week

5. Refine, harden, and document all of the code, and publish them online.
Status: in progress
Plan: All three drivers are currently running with no incoming data.
With the bug fixes last week, they are doing fine – except that there
is no network connection to the MCR site.

6. ISSNIP paper
Status: in progress
We are still working on the first draft. We are currently creating
diagrams and tables while working on #7 for event detection section.

ISSNIP Website: http://www.issnip.org/2009/
ISSNIP CFP: http://www.issnip.org/2009/ISSNIP2009CFP2.pdf

7. Event detection using complex event processing engine
status: in progress, on going
UCSD received great use case scenarios from UCSB. We will implement
some of them by the end of May, and the results will be included in
the paper.

8. Wireless options for buoy
Status: on-going. UCSD is surveying how others are using 802.11n in
the outdoor network. If so, what type of conditions that they put the
networked devices? We want them to work in rainy weather conditions
within 2 km range.
Plan: Survey further, and then contact the others with experiences.

Andy will be gone for a couple of weeks after next week. Let's
discuss the next phase deployment plan more concretely next week.

MCR call reminder:
We will have a regular weekly call on Wednesday May 27, 2009 from 11
a.m. to 12 p.m.

Thank you,

Sameer and Peter

Hi,

Here is the summary of the activities from May 2nd to May 8th 2009.

We are aware of the fire in Santa Barbara, and we know that Andy had
to evacuate from home. We hope that everyone is safe, and please let
us know if we can be of help.

NTL update:

NTL and UCSD both ordered new machines for the experiment, and Luke is
following up on the on-ramp and off-ramp codes, and he will
decommission the old NTL machine once the new machine arrives.

NTL QA/QC follow-up call:
Next Tuesday, Tony Fountain from UCSD and Chris Solomon from NTL will
have a call to discuss QA/QC.

MCR update:

The data stopped streaming from Moorea on Thursday since 2:30 a.m.
Moorea time. We identified the problem to be the DSL modem, which is
connected to a wall supplying the unregulated 220 V voltage. We reset
the modem to a factory default mode and tried other methods to recover
from the failure, but it never came back normal - we are certain that
it is broken. Many thanks to Vinny Moriarty from CSU Northridge who
is in Moorea now.

Short-term solution for the DSL router:
Given the broken modem, we need to replace it or ship a new one. Here
is one option.
1. Given the emergency situation, UCSD can purchase the DSL modem and
ship it to Moorea. Please let us know if we should proceed with this
approach.

Long term solution:
As planned, we would need a 220v UPS.

Coral Reef Workshop (May 5 or 12, 2009):
We are planning a coral reef workshop next year. Please check your
schedule and tell us which date works better for you. (May 5 or 12,
2010)

Call with UCSB MCR information manager:
This week, Gastil from UCSB and Peter S. from UCSD discussed various
issues: data usage, web site, data archive and maintenance of system.

Here is the progress update on the list of action items from last week.
0. Write the start-up script in case of reboot or crash:
status: in progress
plan: We want to automate the recovery process from the power failure,
and we will start writing this script next week.

1. Write up the instruction for using RDV for MCR streams.
status: demo is done, but the instruction needs to be written and sent out
plan: Write it next week.

2. Create an example of how to display a graph on a web page,
from the real-time streaming data using DT.
Status: demo is done for Gastil this week, and the instruction needs
to be written up

3. Refine, harden, and document all of the code, and publish them online.
Status: in progress
Plan: Over several network failures, the driver programs for met
station and CTD have been running without any problems. However, Axis
camera source had bugs, and they were fixed.

4. ISSNIP paper
Status: in progress
UCSB and UCSD are collaborating on a paper titled "Digital Moorea:
Cyberinfrastructure for Coral Reef Ecology."

ISSNIP Website: http://www.issnip.org/2009/
ISSNIP CFP: http://www.issnip.org/2009/ISSNIP2009CFP2.pdf

5. Wireless options for buoy
Status: on going
Plan: We will look further into this next week.

6. Event detection using complex event processing engine
status: in progress, on going
We are integrating complex event processing engine with streaming data
middleware for QA and QC. UCSB had two scenarios:
a. coral reef bleaching condition event: if the solar irradiance and
water temperature go above a threshold value, raise an alarm.
b. Salinity checker: if salinity change dramatically without no rain
drops or no sudden change in temperature, raise an alarm.

UCSD will implement these queries in our system, and UCSB will follow
up on defining the thresholds for the queries.

Thank you,

Sameer and Peter
p.s. We welcome Gastil to the mailing list.

Hi,

Here is the summary of the activities from April 25th to May 1st 2009.

NTL update:

As planned, NTL spec’ed out the machine for the test activity and
discussed it with UCSD. Luke will over it in next few days. UCSD
also shared the onramp and offramp code with NTL.

From the weekly call, we had the following actions items:
Luke will place the order in next couple of days.
Luke will take a look at the source code and email Sameer and Peter if
he has any questions.
Starting next week, we will talk about performance tests of the
database system at NTL -- scalability, robustness tests etc.

MCR update:

During the weekly call, we covered the following topics:
- demonstrate how to create a graph
- check the schedule for the next deployment
- discuss ISNIP paper
- Coral reef workshop (MARK YOUR CALENDAR –May 5 or 12, 2010)

We demonstrated how to create graphs in flexible way, and how to embed
them in the HTML: http://iguassu.sdsc.edu:8080/webTurbine/WebPlot.html

We also discussed that our MCR deployment is still on schedule for
summer testing and late fall deployment at Moorea. UCSB folks will
discuss the details (engineering staff and etc.)

Following the last week’s discussion, we continued thinking about the
automated event detections. Peter will follow up on this next week
with Libe and Andy. This will be part of ISNIP paper.

Here is the upcoming workshop information.
Workshop title: Coral Reef Observing: Science and Technology
Date: Either May 5 or 12

PRAGMA and Open Source DataTurbine Initiative are planning to host a
one day workshop on Science and Technology for Coral Reef Observing
Systems. This workshop would be co-located with the PRAGMA 18
Workshop, to be held in San Diego (if possible in the Calit2 building
on the UCSD campus) during the first two weeks in March 2010. Since
the last meeting of the coral reef technology group was March 2008,
holding a meeting to look at status updates, new technologies, and
ways to integrate remote sensing data to “ground” data in 2010 is very
timely. It is envisioned that this one day working meeting would start
with short updates from various coral reef groups that have been
associated with CREON, followed by technology status updates, from
both PRAGMA and non-PRAGMA sites. Finally, we envision a mixture of
working groups and tutorials. We have started contacting potential
participants.

Here is the progress update on the list of action items from last week.
0. Insert the streaming data from all three sensors into DT server
Status: complete
Please use RDV to connect to iguassu.sdsc.edu:3333
You’ll be able to see all the data. It goes far back. Be aware that
it takes some time to load the data from long time ago (last
September).

1. Write the start-up script in case of reboot or crash:
status: in progress
plan: write this script after I finish the above drivers to run them.

2. Write up the instruction for using RDV for MCR streams.
status: demo is done, but the instruction needs to be written and sent out
plan: with screen shots, the instruction will be written up after #1
and #2 are done.

3. Create an example of how to display a graph on a web page, from the
real-time streaming data using DT.
Status: done, needs to write it up

4. Refine, harden, and document all of the code, and publish them online.
Status: in progress

5. ISNIP paper
Status: in progress
UCSB and UCSD are collaborating on a paper titled "Digital Moorea:
Cyberinfrastructure for Coral Reef Ecology." We plan to submit this
paper to The Fifth International Conference on Intelligent Sensors,
Sensor Networks and Information Processing (ISSNIP 2009). Sameer will
contact Andy, Libe, Russ and Sally for the science section. Peter
will follow up with Andy and Libe on event detection section.

ISSNIP Website: http://www.issnip.org/2009/
ISSNIP CFP: http://www.issnip.org/2009/ISSNIP2009CFP2.pdf

6. Wireless options for buoy
Status: in progress.

Plan: The testing process will be staged just like before: test in
UCSB, and deploy it in MCR. We are leaning towards 802.11n or WiMax
because we can bypass the logistical issues with French government by
avoiding radios. We will look into this further next week.

Thank you,

Sameer and Peter

Hi,

Here is the summary of the activities from April 18th to April 24th 2009.

NTL update:
Today, following up on the last week’s homework assignment listed by
Peter A., Sameer and Luke had the call to discuss action items and
dates. The short list for next week is the following:
Luke – spec. out and order a new Point of Presence (PoP) node/machine.
Sameer – send on-ramp and off-ramp code

A new teleconference reminder:
Sameer, Peter and Luke will have a weekly phone call at 1 p.m. every
Friday. Next one is on the 1st of May.

Details on the plan:
Discussed PoP node for TLS
• Buy new workstation
• Send proposed specs to Sammer for OK by 4/27

Order, have new machine should arrive May 15ish
New Machine at Trout Lake Station will have following S/W:
• Loggernet 3.4.1
• Dataturbine
• LoggerNetSrc

Onramp/Offramp
• Sameer will send code to Luke by week of the 27th
• Luke modify offramp for Vega (DataTurbine->LoggerNet) and also Luke
will take a look at the onramp code (LoggerNet->DataTurbine)

Dataturbine
• Data fed into dataturbine at NTL
• Mirrored to UCSD node (Push Mirror Routing)
• Sameer will provide Downlink node details. Downlink node will be
hosted at UCSD).

Database Testing: Luke will lead this and will discuss it with Sameer and PeterS
• Create use-case scenarios
• Develop example queries
• Use example queries and use-case scenarios to devise tests

MCR update:
This week, over the Wednesday teleconference, we discussed the plan
for our next meeting. Instead of a visit, UCSD will have the VTC with
UCSB using EVO from Caltech (for video) and Skype (for audio). During
the call, we demonstrated the RDV and viewed the streaming data from
CTD deployed in Moorea. The streaming data server address is
iguassu.sdsc.edu, and the port number is 3333. The streaming data
plus other data will be stored in the RDBMS. For DB schema, Andy
mentioned that Gastil, the information manager, wanted to learn more
about ODM and OGS, and possible visit to UCSD.

We also discussed automated event detections. Andy and Libe will also
work on use case scenarios for detecting scientific events. For
example,
1. detect rain fall, check the salinity
2. monitor solar irradiance, check temperature (?) for bleach conditions

Here is the progress update on the list of action items.

0. Set up the DT server for streaming data
Status: complete, iguassu.sdsc.edu is serving data.

1. Insert the streaming data from all three sensors into DT server
Status: in progress, delayed due to switching to a different approach

CTD: done and mirrored to the streaming server
Camera: retrieving the image is done.
LoggerNet: parsing of the file is done

Approach: instead of WebDAV approach, we will write a driver that runs
and inserts the data at a fixed time interval. LoggerNetSrc will have
some failure mechanisms built-in, and have some options in the
configuration file to operate flexibility with LoggerNet.

Plan:
Finish all of them next week. Send the code of LoggerNetSrc to Luke as well.

2. Write the start-up script in case of reboot or crash:
status: in progress
plan: write this script after I finish the above drivers to run them.

3. Write up the instruction for using RDV for MCR streams.
status: demo is done, but the instruction needs to be written and sent out
plan: with screen shots, the instruction will be written up after #1
and #2 are done.

4. Create an example of how to display a graph on a web page, from the
real-time streaming data using DT.
Status: complete and tutor
Plan: demonstrate it next Wednesday and write it up.

5. Refine, harden, and document all of the code, and publish them online.
Status: in progress

6. Start writing a paper on the MCR deployment
Status: in progress
We wrote the half-page description of our setup in MCR.
Plan: write down more ideas and use cases

7. Start researching wireless options for buoy
Status: in progress. Libe and Andy mentioned logistic issues involved
in deployment, and they should be also researched and addressed going
forward.
Plan: UCSD will research the communication hardware that Kim suggested
last week, and follow up on it.

Thank you,

Sameer and Peter

Hi,

Here is the summary of the activities from March 11th to March 17th 2009.

NTL update:

This Wednesday (4/15/2009), we reviewed our past work and planned the
summer work items through the teleconference using EVO for video and
Skype for audio. Peter A. sent out the detailed meeting notes that
contain action items on the separate email. Please refer to his
“Summary VTC 12Apr09.doc” for further information.

The UCSD sys-admins are still working on the igaussu.sdsc.edu.

MCR update:

Andy, would you please follow up on setting up the meeting time in
UCSB? March 23 or 28th are bad for us.

David deployed the CTD in Stearns Wharf, and the data was flowing to
UCSD. The eventual goal of the CTD in Stearns Wharf was to inform the
deployment in MCR. The deployment was successful, and now the data is
being redirected to its eventual destination (SCOOS?).

Currently, the data from MCR is being streamed to the machine in UCSD.
We report one incident of machine being rebooted due to security
issues related with a web server. We will describe its details on
action item #1. Here is the list of action items reported last week,
and their status.

1. Insert the streaming data from all three sensors into DT server
This action item has three parts to it: writing a driver for CTD,
MetStation and Axis camera. CTD driver is currently working, but the
approach that we took for the other two sensors turned out to be not
secure.

Approach: WebDAV solution

Status: incomplete.

Problem: the planned approach (Apache Tomcat 5.5) had security problems.

Plan: Complete it by next week using an alternate approach.

For Axis camera and met station, it would be easy to insert the data
points using a file system like approach, provided by WebDAV. As we
learned from the NTL testbed, frequent reconnection would lead to
short amount of data in the DT server. Because of this, Matt M. and
John W. suggested using WebDAV approach. However, this approach
caused the machine to crash.

We did not realize that the machine has rebooted until the next day
(for about ten hours), and the streaming data from the CTD was not
captured. Currently, the machine does not automatically recover from
the reboot; we manually restart the programs (DT server, CTD driver,
LoggerNet, etc.). However, we can automate all this process by
writing a start-up script. The task of writing the script will be
added as an extra item to the task list (as a lower priority one).
This additional script is necessary to minimize the impact of machine
reboots.

new: 1.2 Write the start-up script.

2. Write up the instruction for using RDV for MCR streams. UCSD will
set up a meeting with David and others.
status: not yet (due to iguassu not being ready)

3. Create a diagram of the MCR deployment from sensors all the way to UCSD.
Status: complete, see the email attachment

4. Create an example of how to display a graph on a web page, from
the real-time streaming data using DT.
Status: in progress.
Plan: due to security issues, we will use Linux machine
(iguassu.sdsc.edu) and look into securing the server.

5. Refine, harden, and document all of the code, and publish them online.
Status: in progress

6. Start writing a paper on the MCR deployment

7. Start researching wireless options for buoy
Status: Libe initiated the conversation with researchers from OSU. We
exchanged emails with them, and we will further explore various
options.

MCR call:
As usual, we have a Wednesday teleconference call scheduled at 11 a.m.
on the 22nd of April. Please let us know if you want to do VTC for
this one.

Thank you,

Sameer and Peter

Hi,

Before we begin the status update, let’s decide whether UCSD will visit UCSB in short future (within this month), and exactly what date that will be. At the meeting, we will plan for the next deployment, future direction and its timeline. For UCSD, April 23rd and April 28th are not good dates so far.

Here is the summary of the activities from March 4th to March 10th 2009.

NTL update:
We are further conducting the test on push mirroring feature of the DT using PRAGMA grid machines, and we are closely collaborating with Erigo.
iguassu.sdsc.edu is being reformatted, and will now be managed by system administrators at UCSD.

NTL teleconference reminder:
We have a teleconference next week, Wednesday, April 15, from 10AM to 1PM WI time (8AM-11AM CA time).

MCR update:
Overall, the deployment is now a success. Since 6 p.m. PDT today, we have:
1. all three instruments up and running with data being sent to UCSD
2. data from CTD being inserted into DT and being archived to a file
3. remote control of sampling and output using MOXA and VPN from the router
Around 6 p.m. P.D.T., Andy finalized fixing the cable and deployed the CTD. Since then, UCSD remotely configured the output format of the CTD and started getting the real-time data from it. Earlier in the week, UCSD troubleshooted the remote network configuration using VPN and through the remote access to the router. In summary, we currently have full access to the system from the US.

This week, we started working on the following items:
1. Insert the streaming data from all three sensors into DT server (will be done next week)

2. Write up the instruction for using RDV for MCR streams. UCSD will set up a meeting with David and others (it’s used for checking the real time data streams).

3. Create a diagram of the MCR deployment from sensors all the way to UCSD (will be done next week).

4. Create an example of how to display a graph on a web page, from the real-time streaming data using DT (will be done next week)

5. Refine, harden, and document all of the code, and publish them online.

6. Start writing a paper on the MCR deployment

7. Start researching wireless options for buoy

Regarding the wireless options for buoy, Brian Emery described the wireless communication devices and their settings in Channel Islands and UCSB. We will follow up on the email, and evaluate its technology for MCR station.

MCR call:
I don’t know if we should set it at 11 a.m. this week because of the NTL call from 8 to 11. We should push it to at least 11:15 a.m. What do you guys think?

Thank you,

Sameer and Peter
p.s. We welcome Scott for joining our mailing list.

Hi,

Here is the status update from March 28, 2009 to April 4, 2009.

MCR deployments: success (with a hardware failure that will be fixed).

We had three technical objectives:
1. stream the data from the CTD
2. stream the data from weather station
3. configure Axis video camera

Because of power fluctuation and outages, we wanted to avoid having a local computer for any of the three tasks (side note: a local computer died this week). We tested out the power failure and network failure modes as well as dynamic IP changes prior to this trip. The tested configuration was adopted in Moorea, and it all worked. (well, sort of until the CTD cable got broken).

Problem: I broke the custom-made wire for CTD by moving the cable around. Near the very end of the cable (where the serial DB9 connector is), three lines were intertwined. The RJ45 cable was soldered onto the DB9 male connector line with the external power supply line. Somehow the soldering part had some bad connection, and I broke it. See the attached pics.

Solution: Andy and Keith will ask one of the electrician to remake the ending of the wire.

Side note: In order to isolate the problem, we pulled the CTD out of the water, tested the seabird with a different cable to confirm that it was the wiring issue.

Plan: We will fix the cable, and put the instrument back in the water.

So, on the objectives:
1. seacat driver used to work when the cable was good, and the screen shot is attached.
2. LoggerNet software is running in UCSD, and the screen shot is attached (COM4 is for MCR, and coll state = normal is good)
3. Axis camera is configured to view from the outer world, and the screen shot is attached (it's night now).

Plan: before leaving, I'll finish writing the document explaining the configuration of each devices. After we get back, we will do the following:
1. write up a more detailed report
2. stream in the LoggerNet data into DT
3. stream in the video camera images into DT
4. wait for the ctd fix, and stream in the data.

Thank you very much to everyone who helped this deployment to be a success.

Cheers,

Sameer and Peter

Hi,

Here is the update from Moorea on MCR deployment from March 21, 2009
to March 27, 2009.

Tony and Peter Shin made it safely to the Gump Station in Moorea on Tuesday.

Here is the progress that we made so far:

Network connectivity - reconfigured and working.

Streaming the data from Seacat 16plus - MOXA boxes are configured
correctly to stream the data in, and the data is being captured into a
file using the driver.

Here were some issues that we faced:
Network - dsl modem was not connecting to the internet, and the router
was not configured properly.

Driver for Seacat 16plus - We were hoping the CTD in MCR will match
the setup in UCSB identically. There were two differences:
- 1. long network delay (ping time: 700ms)
- 2. data format was different (in hexadecimal instead of decimal)

-1. We did not expect that long network delay would affect serial
protocol. For Stearns Wharf, we did not need to turn off the
FlowControl bit from the serial protocol in our driver. However,
because we there is a huge delay in the tcp packet (700ms) , the
FlowControl bit for Serial protocol was unnecessary - actually it was
hindering the data flow. So, we turned the Serial FlowControl bit off
in our driver, and we are using TCP layer to do a flow control,
instead.

-2. For data format, David S. quickly helped us change it to decimal
numbers, but the number of variables were not the same as ones from
UCSB. We'll discuss this issue with Andy in coming days.

Plan:
For seacat 16plus, we will stream the data into a DT so that we can
see the data from Moorea using RDV.

For weather station, we used the RJ45 to DB9 jack on both ends to
connect it to a local computer. After testing its connection, we will
stream the data to UCSD.

The camera has a power flowing in since it has a light on it, but it
needs to be reset. Andy and I will cooridnate next week on this, and
stream the image into UCSD next week.

In addition, we are planning various performance tests as well as
reliability tests.

Cheers,

Peter, Sameer, Tony
p.s. We welcome Derik in our mailing list!

Hello,

Here is the status update from February 28 to March 6.

NTL testbed:

This week, we conducted the similar push-mirror test on PRAGMA grid
with one sample per minute instead of one sample per second. Similar
to the last week's result, after one hour of disconnect time, the
reconnection happened within a minute. However, unlike the last
week's result, twenty minutes worth of data were lost instead of three
minutes. Upon finding this bug, Erigo fixed the problem by shortening
the loss gap from twenty minutes to one minute.

Here is the detailed setup of the experiments before and after the fix:
------------------------------------------------------------------------------
Before the fix:
OSDT servers: upstream server located at machine in Zurich and
downstream on my laptop at UCSD.

Source: Machine located in Zurich. This machine ran my OSDT source
program. Here is how the source program was run:
java -jar OSDTBenchmarking.jar localhost MonotonicLongSource
SequenceID source 0.01666 6000 1

Sampling frequency : 1 sample/min

Sink: RDV was used to inspect data loss.

disconnection period: 1 hour

disconnection start: 1:38 pm PDT
connectivity reestablished: 2:38 pm PDT
delay to reestablish mirror connectivity: 1 minute (at 2:39 pm PDT
data started flowing in from Zurich to UCSD).

observation: Data for the first 20 mins (from 1:38 to 1:58 pm PDT) of
the disconnection period was lost. No other data loss was observed.

After the fix:

OSDT servers: upstream server located at machine in Zurich and
downstream on my laptop at UCSD.

Source: Machine located in Zurich. This machine ran my OSDT source
program. Here is how the source program was run:
java -jar OSDTBenchmarking.jar localhost MonotonicLongSource
SequenceID source 0.01666 6000 1

Sampling frequency : 1 sample/min

Sink: RDV was used to inspect data loss.

disconnection period: 1 hour

disconnection start: 12:34 pm PDT
connectivity reestablished: 1:34 pm PDT
delay to reestablish mirror connectivity: 1 minute (at 1:35 pm PDT
data started flowing in from Zurich to UCSD).

observation: Data for the first 1 min (for 12:35 pm PDT) of the
disconnection period was lost. No other data loss was observed.
-------------------------------------------------------------------------------------------

Fix details: Fix does a round-trip confirmation of the receipt of each
mirrored data frame. It makes sure that each mirror frame is sent to
the downstream server. Thus there may be some impact on high frame
rate performance. Fix could mirror over 60 data frames per second.
There is an option to turn off the synchronization if performance is
more important than reliability.

Conclusion: The fix seems to work very well and data loss is reduced
to just 1 min. worth of data. We're pursuing further avenues to
further reduce that if possible. We're going to conduct more tests of
the updated DataTurbine server code with various sampling regimes and
disconnection period.

MCR Update:

Three days prior to the trip to Moorea, we tested all the critical
hardware and software pieces. The dynamic IP test over nine days
shows that our system recovers from the change with loss of 1.5
minutes of data, and the data are logged using the driver in a flat
file. We picked up the long rugged serial cable line from the
Teledyne Company this week. We tested all the RJ45 to db9 male and
female plugs to fit with the cables. We tested all the MOXA boxes for
Moorea.

Over the weekend, we will finish up the integration of DT part with
the driver, and configure the MOXA box to receive data from Moorea.
Also, all the items will be packed.

Today, David Salazar replaced the batteries in the Seacat 16plus in
UCSB. We will deploy the Seacat 16plus and the camera during the week
of April 6th to 10th, probably mid-week in Stearns Wharf.

UCSD returned the Seacat 16plus to SIO today.

Andy, if the research permit does not come through by Sunday morning,
please send us the alternate instruction by Sunday afternoon.

Thank you,

Peter Shin and Sameer Tilak

Hi,

Here is the status update from February 28 to March 6.

NTL testbed:

We have started conducting test for push-mirror reliability on PRAMGA
grid. The early test shows that it loses about three minutes of the
data when it first gets disconnected. However, no other data is lost
when the connection came back alive after sixty minutes of network
outage.

Here was the set up:

OSDT servers: upstream server located at machine in Zurich and
downstream on a laptop at UCSD.

Source: Machine located in Zurich. This machine ran my OSDT source
program. Here is how the source program was run:
java -jar OSDTBenchmarking.jar localhost MonotonicLongSource
SequenceID source 1 6000 1

Sampling frequency : 1 sample/sec

Sink: RDV was used to inspect data loss.

disconnection period: 1 hour

disconnection start: 3:18 pm PDT
connectivity reestablished: 4:18 pm PDT
delay to reestablish mirror connectivity: 1 minute (at 4:19 pm PDT
data started flowing in from Zurich to UCSD).

observation: Data for the first 3 mins (from 3:18 to 3:21 pm PDT) of
the disconnection period was lost. No other data loss was observed.

Pinging machine in Zurich from UCSD to figure out RTT value and here
was the outcome:

64 bytes from 130.60.144.225: icmp_seq=0 ttl=47 time=185.731 ms
64 bytes from 130.60.144.225: icmp_seq=1 ttl=47 time=185.647 ms
round-trip min/avg/max/stddev = 185.647/185.725/186.036/0.100 ms

Next week, we'll conduct more test on PRAGMA grid.

MCR update:

We are wrapping up the hardware tests between UCSD and UCSB.

With help from UCSB network technicians, we conducted the test that
mimics frequent changes of IPs in MCR. Every thirty minutes, the IP
was expired for the router while MOXA box had a static IP. Although
the set up we needed was a modem with changing IP, this was the best
experiment we could conduct. We found that when the IP change occurs,
some data gets lost, but it recovers from it quickly within two
minutes. The bad data looks similar to the ones seen in other tests,
and they will be tossed out in the parser.

The only uncertainty (regarding the hardware) currently known to us is
the UV-resistant, weather-proof long serial cable. Ideally, we would
like to test the long serial cable with CR1000, but this might not
happen before the trip. David is talking with the company on creating
a customized serial cable in the meantime (please refer to his email
this week for the details).

I have another idea though. We currently have ethernet (RJ45) to DB9
converters (both female and male ones) in the lab. Using them, we
might be able to use the current setup without the long serial cable.
Andy or Keith, please correct me if I'm wrong:
- currently in MCR, there is a UV-resistant ethernet cable that
connects the weather station to the computer in the lab.

Right? Then, with the RJ45-to-DB9 connectors, we might not need the
serial cable. This way might be the easiest if it works. From the
wiki site from the last week
(http://en.wikibooks.org/wiki/Serial_Programming/RS-232_Connections),
we found that the cat 5 cable is good for 150 m. So, there is a good
chance it will work. But, let's not take the chance, and let's
procure the long, rugged serial cable anyway.

We obtained the IP for MOXA boxes and the router: dynamic for the
router and static for MOXA boxes. We will set them up and test them
next week. Towards the end, we'll pack them to go (while one of them
will be set up the receive the data from Moorea).

We begun integrating the components towards end of this week, and they
will be tested next week.

MCR teleconference:
We have a teleconference on 3/18/2009 from 11 a.m. to 12 p.m.

Cheers,

Peter and Sameer

Hi,

Here is the status update from February 28 to March 6.

This week was Paul's last week. Paul has administered Axis cameras
and other servers - we worked on various transition items including
web-dav for cameras and web plug-in.

NTL testbed:
This week, we focused more on transition items rather than actual
tests. The transition items included administration of NTL testbed
servers: osdt-server, iguassu and niagara.

MCR:
We continued testing various pieces of hardware.

Particularly, we tried simulating dynamic IPs for MOXA boxes. and
tested the network failure with 64kB buffer in the MOXA box. If the
network fails, MOXA box takes a few minutes to realize its failure.
During this time, it loses the data for about 3 minutes, but after
that, it starts caching the data into a local memory, and when the
connection comes back, it sends back the data. As seen on the power
failure modes, some data is ill-formed by the network failure. These
will be handled the same way - (1. save the raw-error data into a
file, and 2. do not put them in DT).

Because UCSB assigns the IP address to a particular mac address for 24
hours, it is difficult to simulate changing the IP address of the
router (while keeping the same IP for MOXA). We are currently
conducting the test - we are taking down the router off the internet
for 24 hours. For Moorea, it changes every 5 hours or so, and the
network outage time is much shorter than 24 hours. For now, this is
the best test we can conduct.

We also tested the serial cable connection using 100' serial cable in
UCSD, which worked fine.

We found the following useful information from the web regarding the
length of serial cable versus data transmission rate:

from: http://en.wikibooks.org/wiki/Serial_Programming/RS-232_Connections
--------------------------------------------------
Data Transmission Rates

There are physical limits to how far serial data communication can
occur over a piece of wire. When you apply a voltage onto a wire it
takes time for that voltage to traverse the wire, and there are other
unstable conditions that happen when you send a "pulse" down the wire
and change voltages too quickly. This problem is worse as wires become
longer and the frequency (i.e. baud rate) increases. This distance can
vary based on a number of factor, including the thickness of the wires
involved, RF interference on the wires, quality of the wires during
the manufacturing process, how well they were installed... e.g., are
there any "kinks" in the wires that force it into a sharp bend, and
finally the baud rate that you are transmitting the data.
This table presumes a fairly straight and uniform cable that is
typical for most low-voltage applications (i.e., not a power circuit
that uses 110V to run your refrigerator, toaster, and television).
Typically something like a CAT-5 cable (also used for local networks
or phone lines) should be more than sufficient for this purpose.

Baud Rate Maximum Distance (in feet) Maximum Distance (in meters)
2400 3000 914.4
4800 1000 304.8
>>>>9600 500 152.4 <<<<<
19200 50 15.24
The distance limitation can be mitigated. There are "short haul
modems" that can extend this distance to several miles of cable. There
are also telephone lines, or conventional modems, and other
long-distance communications techniques. There are other ways to
handle data in situations like this, and those signals can be
converted to simple RS-232 data formats that a typical home computer
can interpret.
-------------------------------------------------

Andy or Keith, what is the baud rate for Campbell weather station? We
will be safe to use 80m cable if we use 9600 baud rate, as marked
above.

So, the plan for Moorea is the following:

Use long serial cables to connect the following two devices to the MOXA box:
- Seacat 16+ CTD
- campbell datalogger cr1000

By the way, during the teleconference this week, we thought we had
cr1000 in-house, and we wanted to test a long cable with cr1000. We
checked the Campbell dataloggers in UCSD, and realized that we had
cr23x and cr510. We returned cr1000 because it had a completely
different OS from the ones that our collaborators had. We will need
to find another way to test it. Any ideas, guys? Luke, which ones
did you use, and how long of a cable did you use it with before?

Dave will purchase a UV-resistant, long (>80m) serial cable, in the
meantime, we should look for a cr1000 that we can test the cable with.
However, from the above statement, it should work.

We made a small progress on driver. We are testing the integration of
XML configuration part into the driver. We still need to integrate
the other components: DT client, and a parser.

We requested static IPs for the new MOXA boxes in UCSD, and requested
a dynamic IP for the router that will be installed in Moorea. After
obtaining the IPs, we will simulate the network environment in Moorea
in UCSD.

Andy is working on the physical layout and connection diagrams with
Keith currently.

We can't readily conduct the power failure test with UPS because the
UPS was for 220V. We will do this in Moorea. Even if the power goes
off for a short time before the battery kicks in, we should be OK
since MOXA boxes recovers from power failure safely.

MCR teleconference:
We have a teleconference on 3/11/2009 from 11 a.m. to 12 p.m.

Cheers,

Peter, Paul and Sameer

Hi,
Here is the status update from February 21 to February 27.

Before we mention specific items, our team member Paul Hubbard at UCSD
will leave to work on OOI project, starting March 8th. We started to
work on the transition items including web-dav (related to push-mirror
testing in NTL), camera feeds, and etc. Although we'll miss him
greatly, we wish the best for him.

NTL testbed:
We are setting up the push mirror tests mentioned last week. In short
future, we will also set up the push-mirror test with web-dav approach
(for frequently reconnecting application such as LoggerNetSrc driver).

MCR:
Because March 24th trip is approaching fast, we prioritized procuring
the hardware equipments and testing them. UCSD now has an identical
set of moxa boxes for Moorea and a Linksys router. We also procured
various connectors for serial cables, and the cables themselves. We
will simulate the Moorea system in UCSD, and test/configure the
components before we go to Moorea.

We tested MOXA box connections using the driver (not integrated one).
We had 8 tests:
UCSB power off/on
UCSB network off/on
UCSD power off/on
UCSD network off/on

We conducted the four tests in two different modes: one minute outage
and over ten minute outage. As expected, it recovered in every case.
This is good news for us. However, when the power is lost, only
partial data arrives. We will handle this by both saving all the raw
data to a file and inserting only the good data to DT. Here is the
output from the current driver that records all the raw data:

-----------------------------------------------------
# 23.1987, 0.02251, -0.014, 0.0000, 0.0000, 0.0000, 0.0000, 0.1130,
25 Feb 2009, 16:09:07, -2.4214, 12.9, 3.0

# 23.2007, 0.02251, -0.014, 0.0000, 0.0000, 0.0000, 0.0000, 0.1130,
25 Feb 2009, 16:09:27, -2.4219, 12.9, 3.0

# 23.2022, 0.02251, -0.013, 0.0000, 0.0000, 0.0000, 0.0000, 0.1130,
25 Feb 2009, 16:09:47, -2.4223, 12.9, 2.9
2.4318, 12.9, 2.9

# 23.2422, 0.02253, -0.012, 0.0000, 0.0000, 0.0000, 0.0000, 0.1130,
25 Feb 2009, 16:20:27, -2.4319, 12.9, 2.9

Notice 2.4318, 12.9, 2.9 shouldn't be there.
------------------------------------------------------------

As the recorded data shows, power failure results in badly formatted
partial data. We will handle this in software side (in the driver).
Please tell us if you’d like a comma-separated, good-data only version
of a data file. I can create that easily when writing the driver.

The driver development has been very slow this week due to other
activities. We still need to integrate various components.

Next week, we will focus more on writing the driver while continuing
the tests. We plan on doing the following tests:
- Long serial cables
- Power outage with UPS

Dave will find a long serial cable (over 50 meters), and we will test
the power outage with UPS that Keith has in the lab. I also have two
50' cables, and have a seacat 16+ inhouse. I can try to make it as
long as possible.

Andy, you mentioned having a document/diagram/notes with the
description of buildings, wiring and connections. Would you please
send it to us?

MCR teleconference:
We have a teleconference on 3/4/2009 from 11 a.m. to 12 p.m.

Cheers,

Peter, Paul and Sameer

Hi,

Here is the status update from 2/14/2009 to 2/20/2009.

NTL status update:
We coordinated testing the performance of push mirroring with Erigo.
Next week, we will test it in two WAN settings using computers in
Erigo and Pragma grid. In addition, we will also plan out the tests
for inserting data: Web Dav approach and frequent reconnection.

MCR status update:
Given the end of March trip to Moorea, we are continually developing
and testing the software. We also started procuring necessary
hardware: cables, connectors, router, MOXA boxes, and their
accessories. In coming weeks, we will test the cables, MOXA boxes,
network settings and etc. as identical as possible to the ones in
Moorea. Hopefully, when we get to Moorea, it'll be a repeat of the
experiment without any surprises.

Because Dave S. got sick, we were unable to conduct the test between
UCSD and UCSB yet. We will push this once Dave gets back.

Driver development status:
Given no hardware errors, capturing the data into a file has been
running fine (with a small bug?? - noted below) for ten days. We are
currently integrating the following components with some error
handling:
1. serial reader
2. xml configuration part,
3. DT source client,
4. parser
5. file writer

Known bug(?): The sensor is sampling the data every ten second.
Instead of receiving the data every ten seconds, we are getting the
two data points every twenty seconds. We believe this problem is
caused by how JAVA threads are managed in Windows. Just to clarify,
we are not losing any data, but on half the data, there is ten second
delay to DT or a file. We tried to fix it, but given the other tasks,
this bug will have a low priority.

MOXA box status:
We are in the processing of ordering the identical ones for Moorea at
UCSD. UCSD obtained the following static IP for the current MOXA box:
137.110.118.68.

The first version of the driver will be complete by the end of next
week. Hopefully, Dave gets back before then, and start carrying out
the tests.

I've asked Scott Shannon to let us borrow the equipment for one more
week because we didn't have the UCSB-UCSD connection yet. If Dave
doesn't get better soon, we'll test the network and power failures in
the lab with the newly ordered MOXA boxes (not ideal, but better than
no testing).

MCR teleconference:
We have a teleconference on 2/25/2009 from 11 a.m. to 12 p.m.

Cheers,

Peter, Paul and Sameer

Hi,

Here is the status update from 2/7/2009 to 2/13/2009.

NTL status update:
This week, we focused our efforts on preparing for the MCR deployment.
Starting next week, we'll start conducting experiments on
push-mirroring with frequent reconnect and web-dav.

MCR status update:
Last week, we projected we would test the whole system connecting UCSB
to UCSD. Due to the delay in the MOXA box delivery, we do not have
this connection yet. Meanwhile, we developed and tested parts of the
system.

Driver development status:
We are currently developing, testing and integrating software
components. The driver has been capturing the data into a flat file
from the CTD for the last three days. The driver still needs to be
integrated with XML configuration and DT source client parts. We
developed the xml configuration part so that the program can be
deployed easily without recompiling each time in different setting.
We also developed DT source client that connects to a server and sends
the data, but it needs to be integrated with parser and xml
configuration part. We will integrate all the components and finish
developing the driver next week.

MOXA box status:
UCSB received the package. Dave has asked the IT people at UCSB for
the setup to change the IP address of the MOXA box, in the lab. He
will keep us posted on the details.
UCSD also received the package, and we submitted a ticket to get a
static IP based on the mac address.

Moorea:
Once we confirm that MOXA boxes can handle the errors, we will order
the same ones from the Stearns Wharf deployment (a pair of them). Two
weeks ago, we found out that Stearns Wharf has a box that stores some
energy, regulates the voltage and checks the battery life. In the MCR
station, instead of this Seacat box, we will have a battery backed
UPS. According to Keith and Andy, the power failure can last a long
time, and this Seacat box does not have the sufficient amount of
electricity. This UPS is currently sitting in UCSB (in Keith's
office?) Next week, when we test the power failure scenarios, we will
test with this UPS. We will simulate power failure by taking the plug
out from the wall.

Over the teleconference, we discussed having staged deployment. Next
stage will include wireless radio. Similar to the current setup, we
will test it first in Stearns Wharf, and then in Moorea.

Next week, Dave and Peter S. will coordinate on these various tests.

Update with Jerry W. from SCOOS:
We wanted to learn how others have set up the Seacat devices:
hardware, networking protocols, etc. Peter S. spoke with Jerry W. from
SIO regarding a deployment in Scripps pier and one in Huntington
Beach. They both use Verizon wireless card to send the data using TCP
protocol. Similar to us, they used the Serial to Ethernet converter
and then to TCP. The data is then sent to a computer server that
saves the data into ORB and relational database. In both deployments,
they had static IPs. According to Jerry, they used to have dynamic
IPs and faced several problems. They used Dyn-DNS just like us to
solve those issues in combination with the router that can send out
the change in IP addresses. Our router in Moorea has this capability.
So, it was assuring that we are taking a similar approach as others.
They seem to use a different Serial to Ethernet box, and I will find
out this detail from Shannon Scott.

MCR teleconference:
We have a teleconference on 2/18/2009 from 11 a.m. to 12 p.m.

Cheers,

Peter, Paul and Sameer

Hi,

Here is the status update from January 31st to February 6, 2009.

NTL update:
Last couple of weeks, we experimented with push mirroring with
frequent connect and disconnect mode, and found a bug. We can avoid
this bug in two ways:
1. Don't use DT with frequent connect and disconnect. Instead,
use WebDav method.
2. Configure the push mirror in a certain way to avoid having this bug.
For #1, we will start pursuing this method mid-February with John
Wilson and Matt Miller from Erigo. For #2, we will need to write a
short program to setup the push-mirroring condition. Before the
source disconnects itself for the first time, the user must set up the
push mirror. Because LoggerNetSrc disconnects right away after
inserting the data initially, we do not have time to set up the
mirror. We need to either modify the program or write a short program
for initial mirroring. Since writing this short program gives a
better modularity, we will write a short program mid-February as well.

We also looked into ESPER, a complex event processing engine to help
QA/QC process. We followed and modified a sample program. We created
an event if the number of samples received over last ten seconds
dropped more than ten percent compared the previous ten second count.
In the NTL setting, if we expect to have 6 samples every hour and we
don't receive any, we can create an event (no data received over last
one hour). In it, we also calculated average of values (e.g.
temperature) in each event.

MCR update:
Currently, we are focusing our efforts on deploying Seacat 16+ CTD.
Last week, we were concerned about MOXA box's status. This week, we
checked with MOXA company regarding our settings: frequent changes of
IP, network and power failures - they responded that it would work and
recommended a certain setting (pair wise connection). And, we ordered
it. We expect to receive it early next week.

We are trying to create next week's test to be as similar as possible
to the real set up in Stearns Wharf and Moorea. Through this test,
we'd like to reduce the unknowns as much as possible. Dave will set
up the CTD device with a long cable, and he is also finding out to
what Seabird box it is connected. Moorea does not seem to have this
box, but Stearns Wharf does.

On the driver side, we are designing, developing, and testing various
pieces of program for Seacat 16+. Last week, we developed simple
components such as serial communicator and UDP server and client.
This week, we designed and implemented error handling mechanisms for
various APIs. Currently, we are implementing XML configuration reader
while integrating it with other pieces.

We also are trying to find out how others set it up. Peter S. spoke
to Karen Baker from SIO, and found out Jerry was out sick. He's back
but away today. Karen B. will relay the mesasge to him. Maybe early
next week, Peter S. will consult with him on various issues with
deployment.

MCR teleconference:
We have a teleconference next Wednesday (February 11th) from 11 a.m. to 12 p.m.

Cheers,

Peter, Paul and Sameer

Hi,

Here is the status update from January 24th to 30th, 2009.

NTL update:
The machine in NTL was restarted because of Windows patch. After the
reboot, we reset the archive size to be small (=5) to create a good
condition for push-mirroring to work. Next week, we will report on
the performance of push-mirroring.

MCR update:
As mentioned last week, we will use Stearns Wharft as a testbed for
deploying Seacat 16+. In this week's teleconference, we discussed UDP
communication protocol in addition to TCP. Using TCP connection, we
need two MOXA boxes - a sender and a receiver. TCP is more reliable
than UDP because TCP will resend the packet if the sender does not get
the acknowledgement from the receiver. TCP will keep trying sending
the data for a fixed time, while UDP does not care if the packet is
received or not (no ack's). However, UDP is simple to develop and
deploy. We'd need only one MOXA box near the sensor, and let the
computer in UCSD listen to it (UDP capture). UCSB is currently trying
UDP capturing, and UCSD will coordinate with UCSB on the development
and testing.

Regardless of the protocol, the content of the data will look the same
(unless MOXA box adds characters). We are currently implementing a
parser for the data and a command module for Seacat 16+. This week,
we implemented a simple DT Source client, a simple bidirectional
serial-line communicator and a UDP server and a client. Next week, we
will finish developing a parser and a commander. We will also start
looking into error handling logics behind each mechanism. Near the
end of the week, we will start integrating those components to make
the reliable driver for Seacat 16+.

Right now, the biggest unknown is the status of MOXA box. Not only we
need this piece for our deployment, but we also need some time to
experiment with it. Currently, it is not hindering our development,
but we need it critically after next week.

After talking to Shannon Scott, we also initiated contacting other
technical people who have deployment experience with Seacat 16+.
Particularly, I tried calling the technical person (Jerry Wanetick)
from SCOOS project twice (yesterday and today). But, both times, I
got his voicemail. I'll try calling again next week, and if not, I'll
send him an email asking about their setup.

MCR networking issues:
Andy tried setting up the network in MCR station, and experienced many
troubles. UCSD will double check the settings and contact Andy for
further recommendations and questions.

MCR teleconference:
We have a teleconference next Wednesday (February 4th) from 11 a.m. to 12 p.m.

Cheers,

Peter, Paul and Sameer

Hi,

Here is the status update from January 17th to 23rd, 2009.

NTL update:
The two-way pushing routing has been working fine.

As mentioned in the last week's report, we found a bug in "no-loss"
push mirroring. We experimented with various parameters and reported
the bug to the DataTurbine core engine developers, Matt Miller and
John Wilson.

This bug happens only with large archive and cache size. While they
investigate the bug, we can change the parameters to be small, and get
it working. Next week, we will restart the two-way push routing with
small cache and zero archive. Ebbe made it work under this setting in
the past.

MCR update:
We initially viewed UCSB Stearns Wharf as a testbed activity, that
will help MCR deployment. Through Stearns Wharf experience, we will
figure out as much unknowns about the deployment as possible to make
the MCR deployment robust. We'd like to try a few ways to stream in
the data. In the short term, we will focus on using
serial-to-ethernet device to get the data stream in. However, it can
lose data if the network goes down. By putting a small computer
(gumstick - a computer size of a pack of gum) next to the sensor, we
should be able to attain all the data. However, it will require more
development time, and we expect that this approach has more unknown
factors than the initial approach.

Here is the current status of the deployment.

Network device status: (Serial to Ethernet)
We decided to use MOXA box for ethernet to serial conversion.
Currently, UCSB ordered two; one will be shipped to UCSB and the other
to UCSD.

Seacat Driver Status:
We expect to meet the mid-February deadline.

Seacat 16+ is in our lab now. We borrowed it from Shannon Scott.

We discussed our goal, and physical and network constraints. He
pointed out two other groups using the CTD under the slightly
different constraints. One group had a reliable network connection
with static IP. This group only communicated with the device one-way.
The other group used a cell phone line, and used bi-directional
communication.

Software development progress: Per discussion with other DT
developers, we are using jython to speed up the development process.
We were concerned that custom package such as DataTurbine might be
difficult to import to jython, but we tried and it worked.

We expect a speedy development next week.

Cheers,

Peter, Paul and Sameer

Hi,

Here is the status update from January 10 to January 16, 2009

NTL update:
The NTL box was restarted due to Windows update. The server at UCSD
(iguassu.sdsc.edu) has been down because of physical move of the unit.
They all came back up online. During the downtime of the
iguassu.sdsc.edu, the driver safely archived the data in the local
file, and uploaded the archived data to the server when it came back
online.

Update on no-loss mirror:
We identified a problem with no-loss mirror when the source with big
archive and cache frames frequently connects and disconnects. We are
engaging Matt Miller and John Wilson from Erigo to investigate the
problem further.

MCR update:
Trouble with serial-to-ethernet device:
Using Lantronix box, UCSB (David S.) and UCSD (Paul H., and Peter S.)
tested the failure scenarios extensively. Lantronix box never
recovered from power failure at UCSD. Both NASA and NTL independently
recommended another product, MOXA box for this purpose. They had
occasional power and network failures with MOXA boxes, and they
confirmed that the MOXA boxes recovered the connection without any
problems. Thanks to Larry Freudinger and Luke Wilson on the
thoughtful comments on these matters.

Seacat Driver:
We are still in the designing process. We are spending more time in
design, and hope to spend less time in development. Despite longer
design time, we do not expect any delays on the final deadline of
mid-February for this effort.

This week, we made a few design decisions:
- use the autonomous mode for retrieving the data.
- send in occasional clock resetting command during which the device
will stop sampling (we expect this to be a couple of minutes every
month) - we will be able to adjust this interval.
- downloading the data remotely from the flash rom - not sure if we
should include it or not. Downloading can take hours, and network is
not guaranteed to support this long process. We also need to look at
how much data we are losing through a real deployment.

We also initiated conversation with Chris Jones and Derik Barseghian.
They both developed DT drivers for various devices. We will discuss
collaborative development of device drivers on Feb 16th. We share
similar ideas on future developments.

Chris Jones has written a version of Seacat 16+ driver:

https://bbl.ancl.hawaii.edu/projects/bbl/

He recommends us not using his driver because of particularity of some
functions. Nonetheless, his experience and ideas influenced our
design decisions.

Reminder for MCR teleconference:

Next Wednesday (1/21/2009) from 11 to 12, we have a teleconference.

Thank you,

Peter, Paul and Sameer

Hi,

Here is the status update from January 2 - January 9, 2008.

NTL:
2-way push routing has been working fine.

We are experimenting with push-mirroring for LoggerNetSrc. This is
related to the issue of connecting and disconnecting frequently. Ebbe
solved this problem by setting up a mirror with short cache and
archive for source, and long cache and archive for mirrored source.
He also started mirroring right from the start. Our experiment is
slightly different in that it has long cache and archive for the
original source. In addition, we did not start mirroring from the
beginning. We started mirroring after it has run for many days. We
are noting that it does not behave the way we expected. Here is the
setup. LoggerNetSrc has been running for days. We set up the
push-mirror to a different machine starting with the oldest data.
Then, only one datum exists in the mirrored source. We will replicate
this experiment and report our findings in rbnb-dev. Also, we will
try the same setup as Ebbe's, which is known to work.

MCR:
MCR LTER site:
Andy has been configuring network parameters for various pieces.
Rainy and windy weather in Moorea hindered setting up the Lantronix
box near the roof.

UCSB and UCSD:
David Salazar, Paul H. and I set up the two Lantronix box on each end.
On the UCSB end, David S. connected the Seacat 16plus to Lantronix
box. In UCSD, we connected the Lantronix box to computer through
USB-to-Serial device (Keyspan). The two devices communicated
successfully - Paul and I could view the data, streamed from UCSB.

Then, we tested four different failure scenarios:

1. power failure at UCSB by disconnecting power supply for 1 min 30
seconds to Lantronix box
2. network failure at UCSB by disconnecting the Ethernet cable for 1.5 min
3. power failure at UCSD
4. network failure at UCSD

When the two boxes recovered the connection, it took usually about 3
mins. In case #2 and #4, the connection was never recovered. Being
puzzled, we found from Lantronix website that new firmware fixed
reconnection issues.

After updating the firmware, we also enabled clearing the buffer upon
active and passive reconnect on UCSD end. We tested all four possible
scenarios, and it cleared all four tests. Thinking that firmware
fixed the reconnect problem, not the parameter to clear the buffer, we
disabled the parameters.

However, when we disabled the parameter, it never recovered from #4.
We tried power cycling both devices, but they never recovered the
connection until UCSD and UCSB both enabled clearing buffer
parameters. We will test #1 through #4 again next week with such
setting.

We also note that we did not specify the remote IP and remote ports in
Lantronix box at UCSD because it only accepts numerical IP (no Dyndns
name). Since IP changes frequently in Moorea, we are properly
preparing for the deployment. However, the Lantronix box in UCSB has
proper remote IP and remote ports from UCSD.

If possible, we should also test the connection by changing IP
addresses from UCSB.

Seacat 16+ driver:
We are reading the protocol specification at the moment. We are
experimenting with jython to write the driver to reduce the
development time without losing performance and robustness.

Reminder for MCR teleconference:
We have a weekly teleconference next Wednesday 11-12.

Cheers,

Peter, Paul and Sameer

Hi,

Happy holidays to everyone!
Here is the status update from December 13 to December 19.

Roger Proctor's visit
Before discussing specific updates on NTL and MCR, we had a visitor
Roger Proctor from Australia. Our collaborator Scott Bainbridge
introduced us to Roger, who is a director of IMOS eMarine Information
Infrastructure. He is in charge of collecting, combining and
disseminating the sensor data collected in IMOS network where in Scott
is a director for one (Great Barrier Reef Ocean Observing System
(GBROOS) of the five offices.

We discussed his mission as well as our mission. Technologically, we
showed him a demo of DataTurbine using RDV. Tony and Sameer explained
various projects that we worked and are working on. The sampling rate
that he requires is every ten to fifteen minutes. Although he did not
need our perception of real-time, he liked the idea of having RDV
because he wanted to visualize the data stream from various regions of
Australia in synchronized manner.

NTL update:
iguassu.sdsc.edu is back up again. We restarted the DT server, but we
could not determine the cause of server failure. We saved the log.
We are preparing a final report on the NTL testbed.

Update on no-loss mirror:
Matt Miller and John Wilson have been working on the no-loss mirror.
They sent us a detailed setting for testing environment. We have an
upcoming teleconference with John Wilson at 11a.m. We'd like to
extend the test across internet since they tried in local networks.

MCR update:
Andy set up the network in MCR station. We could ping the MCR
station, but the RDC and other services were not available. Paul H.
diagnosed and sent the solution to the network problem in a separate
problem - it was a subnet mask related problem.

UCSD is still waiting on the static IP from the UCSD network operators.

Development plan for Seacat driver:
I attached a diagram that shows the data flow. Basically, the program
communicates with Seacat 16+ using serial communication line through
predefined commands, and the responses can be interpreted and inserted
into DT server.

Here are the components in the driver:

Serial communication:
I found various sources for serial programming. I'll use the
following libraries/references/protocol specifications.
Serial programming
http://en.wikibooks.org/wiki/Serial_Programming:Serial_Java#RxTx

Java serial interface (JAVAX)
http://www.rxtx.org/

comm serial api
http://java.sun.com/products/javacomm/

Command & Interpreting the response:

Question to Andy and others, do you have any preferred way of querying
the data? I see pros and cons in both ways.

Two ways of retrieving/querying the data:
1. Autonomous – This way assumes that the device is always
connected to the computer. The data is pushed out to the computer at
a fixed interval by Seacat 16+ itself.
- con: if the connection gets lost, the reconnection must
happen with the new IP.
If the data is pushed out during the disconnect,
then it must sent a different query to get the data from flash rom.
The sampling interval relies on seacat 16+ clock,
which drifts quite a bit from our experience
- pro: if the clock is correct, we can have near perfect
sampling time.
The clock can be reset periodically.

2. Polled – computer invokes a query and receives the data.
- pro: It is less affected by the network connection
because the computer connects to the device only when it wants to
query the data.
Since computer clocks can be far more accurate
than the one in Seacat, the timing can be more accurate.

- con: if the network fails at the time of the command,
then it has to wait until it finds the network connection. This
results in loss of perfect sampling rate.

How to handle clock drift in Seacat 16+
- We have seen a long clock drift over 6 months. We should
occasionally calibrate it (once a week?).

We will need to write a parser for responses from Seacat 16+.

The other components include:
- DataTurbine source client
- Time formatting
- Error handling

I expect that this will take about 1 week to design, 3 weeks to
develop and 1 week to test and deploy. So, we expect about 5 weeks.
It should be noted that we do not have a local Seacat 16+ sensor to
play with. The debugging process can be more tricky than usual.

See you next year!

Thank you,

Peter, Paul and Sameer

Hi,
Here is the update on NTL and MCR from December 6th to December 12th.

We will report some problems, findings and solutions on NTL testbed.

Our server at UCSD (iguassu.sdsc.edu) started having problems this
afternoon. I was working with a remote collaborator, and the server
started refusing the connection. We'll get it back up soon and
troubleshoot the problem.

NTL:
We have several updates this week.

Before describing anything technical, this week, we helped Luke
migrating the code repository.

Machine at NTL:
Windows XP patches were applied and the machine at NTL was rebooted.
Some of you wanted to know what happens when the machine reboots.

Here we go:
- Machine starts.
- Before anyone logs in, the machine does not start any processes.
- When a user logs in, startup scripts are initiated. Log-in
automatically starts the DT server and LoggerNet software. LoggerNet
software invokes LoggerNetSrc drivers to DT server every five minutes.
Situation normal.

This scenario has one problem. If no one logs in, then the programs
will not start. We can solve these problems in two ways:
1. auto-login to a default user with proper scripts as startup process.
2. Writing a windows script to coordinate this:
http://www.bleepingcomputer.com/tutorials/tutorial44.html

Update on no-loss mirror:
This week, an original developer of DT, Matt Miller reported his
experiments with the current mirroring capacity of DT no-loss mirror.
Similar to our previous experience, he found that current version of
DT supports no-loss mirroring pretty well - as long as there is enough
memory and disk. Upon a proper setup (push-mirror with append mode),
it can recover most of the data. However, it was not perfect. The
loss of the data happens for about 20 seconds at max in his
experience. The mirrored server thinks that it is still connected
instead of realizing that the connection to the client is lost. He is
looking into correcting this problem.

Update on experimenting the memory and cache size with two-way routing:
Two-way routing mechanism does not have any problems in terms of
mirroring -no loss of data during the transfer. One downside of this
routing is that the old received data disappears (it looks like the
cache size wasn't sufficient.) it connects and disconnects every
five minutes, and such reconnection makes the old data disappear It
turns out that exploring this relationship was done before. Ebbe
tried the same routing scheme, and made a work-around to obtain all
the data (by setting up a local mirror).

We experimented with four different cache and memory sizes (in orders
of magnitude difference). We used RDV to view the data, but their
sizes did not make much difference. According to Matt Miller, because
DT was designed for a quick reconnect - fileset handles runs out if
many reconnection occurs. We do not expect to see a change in this
behavior from DT since it implies redesigning the internals of DT. We
will have to use the work-around if we want all the historical data in
DT.

MCR:
We are pushing ahead with the deployments. We currently have two
parallel efforts of the same deployment: one at Stearn's Wharft and
one in Moorea. We had a teleconference this Tuesday, December 9th,
and discussed various progress and problems.

Stearn's Wharf near UCSB:
It already set up the secure box with IPs and serial ports.

Moorea:
There are three devices: Seacat 16+, met-station and video camera.

Seacat 16+: It's connected to the internet using Lantronix box.
Met-station: Because Andy had a wrong part to connect the met-station
to a secure box, he directly connected the met-station to a computer
through a serial port. He also set up all the port to the router.
Video camera: It's powered, but not set up yet.

UCSD:
We have the secure box with dynamic IP, but no static IP yet. We set
up the serial to USB for secure box. We are also looking at the
document of SeaCat 16+ to come up with the development plan for the
robust SeaCat 16+ driver. We will send out the development plan next
week.

For MCR, we won't have a teleconference until next year.

Any questions and comments are welcome.

Thank you,

Peter, Paul and Sameer

Hi,
Here is the status update from November 22nd - December 25.
For MCR, both UCSD and UCSB worked on getting hardware equipments in
place. For NTL, we observed occasional network failure.

NTL:
We deployed a new version of LoggerNetSrc that records network or
server failures. Since our deployment on November 11th, we have seen
the network or server failures on the following time: (they were
network failures since the server has been up)

Wisconsin time (Central Standard Time):
2008-11-20 23:00:26
2008-11-24 20:40:26
2008-11-24 21:00:26
2008-11-26 01:20:38
2008-11-27 11:50:31
2008-12-05 13:30:31
Otherwise, the data has been flowing fine.

MCR:
Everyone's installing, testing and deploying the hardware for the
first deployment.
Due to this hardware-related work, we postponed our teleconference to
Tuesday at noon next week. I'll send out a reminder on Monday.

In the separate email, Andy described six technical problems he faced
and resolved when he got back to Moorea.
1. power to met station
2. asdl router power
3. SENA serial-to-ethernet adaptor
4. DYN-DNS
5. keyspan
6. windows xp patches

Cyril and Dave worked on network device connecting issues. Dave
installed ethernet to serial devices to the router so that it can now
communicate with CTD devices. Later on, one of them will be
configured for MetStation (LoggerNet). In UCSD, we checked the
installation of LoggerNet software to be 3.3.1 in the Windows box
named yunyi.sdsc.edu. We set up the USB-to-serial device and will set
up the serial to ethernet (Lantronix secure box) next week. We
updated the figure with the correct name for PC.

Next week during the teleconference (Tuesday 9 at noon), we will
discuss hardware/networking/technical issues. Paul H. suggested
adding iBoot device near the sensors. iBoot has an IP, and it can
shut off the power remotely, and power it again. We should discuss
where it might be useful in the diagram. Since there is no objection
to google doc as repository, I'll create it there. I started
uploading files there. Once it's set up, I'll send the invites.

Thank you,

Peter, Paul and Sameer

Hi,

Up until last week, we wrote two separate weekly reports: North
Temperate Lakes(NTL) from U of Wisconsin and Moorea Coral Reef(MCR)
from UCSB. Because most efforts apply to both groups, we will
generate one report starting this week.

NTL:
We are wrapping up the testbed activities for NTL. Two-way routing
mechanism is up and running, and there has not been a network or
server failure since we deployed the new version last week. This
LoggerNetSrc will be used in MCR deployment as well.

MCR:
We discussed the list of hardware that will be deployed in Stearns
Wharf, Santa Barbara. We drew a diagram that shows specific model
numbers and the type of network connections. This jpeg image is
attached in the email. However, the attached image does not contain a
camera in the deployment package. We currently have a camera
identical to the one installed in Moorea. Would you like to put this
camera in Stearns Wharf as well? We also have the enclosure for it.

When all the hardware gets installed in place, we can easily deploy
LoggerNetSrc and stream in the data from CR1000. The software
component is already developed and tested. What is not understood is
how the errors are handled by network-connection converting devices
such as SecureBox(Ethernet to serial) and Keyspan (serial to USB). We
will need to experiment with them.

Although we developed a prototype version of Seacat driver, it was not
robust. In order to estimate the timeline for developing the driver,
we need to understand the serial protocol specification as well as
data format. We will look into the documentation next week and start
estimating it.

A reminder for MCR:
Teleconference scheduled for Tuesday, November 25, at 1 p.m.

All comments and questions are appreciated.

Thank you,

Peter, Paul and Sameer