Quake Data Distribution System (QDDS)

Version 1.3.3 Level 2004.11.08

Contacts
Alan Jones, SUNY Binghamton
David Oppenheimer, USGS Menlo Park

Contents

Revisions

Back to top

Introduction

The Quake Data Distribution System (QDDS) provides a method for distributing earthquake data over the Internet. While similar to a classical client-server system, it has some important differences. We refer to it as a hub-leaf system. Hubs send event information to attached leaves. In addition to receiving information, leaves can originate information and send it to one or more hubs. These hubs then send to their leaves.

The system is written entirely in Java which makes it platform independent. It has been tested on Solaris running on a Sun SPARC and on a PC running Windows 95, 98, NT, 2000, or XP..

The initial QDDS System was written by Stephen Jacob who was a summer employee at the USGS Menlo Park Laboratory in the summer of 1998. Further development was done by Alan Jones. Read the documentation file: QDDS.notes.html to understand how the system works.

QDDS is used to produce merged catalogs using QDM Quake Data Merge program (formerly CNSSM for the Council of the National Seismic System Merged program). For more info please contact David Oppenheimer at oppen@usgs.gov.

Back to top

Installation

Installing Java

You must install Java before installing QDDS. Install the Java Software for the Desktop (Java VM) release 1.3 or later

Note: If you want to make modifications to QDDS, you will need the Java Development Kit available at the same site below. To just run the package, all that is required is Java VM.

Select JRE from:

http://java.com/

Pick the download for your environment, either Windows32 (Windows 95, 98, NT, 2000, or XP) or Solaris and install.

Download QDDS files

You only need three files to use QDDS:

QDDS.jar - the Java executable
QDDS.config - Configuration file
comm.lst - definition of network connections

All the files are contained in the latest tar file located in the QDDS "src" directory. Untar/extract the files into a local directory.

Installation Over an Older Version

Just download the executable QDDS jar file

Back to top

Setting up QDDS

The default QDDS.config and comm.lst files are configured to be ready to use. They will set you up as a transient leaf and will connect to you the system defined in comm.lst. If might be a good idea to test the system with this default configuration. To do this:

java -jar QDDS.jar

After a few minutes you should start getting alive messages from the qdds1 and geo-www hubs and once in a while an actual earthquake message.

After you verify that it runs, you will want to configure for your specific setup. To do that, you must edit the two files: /p>

comm.lst

QDDS.config
/pre>

The comm.lst file is a list of the sites to which you want to attach. If your computer will be a hub, it is a list of your leaves. If your computer will be a leaf, it is a list of your hubs.

Note: This is the normal way things are run. However, there is nothing stopping a leaf from communicating with a leaf and a hub with a hub.

comm.lst

Here is an example comm.lst:

#format -- hostname/IP:password:UDP port: TCP port:e-mail address:Freeform text field:
qdds1.wr.usgs.gov:qdds2Mary:2222:2222:tigger@tsu.edu:From Menlo Park to Mary
qdds2.er.usgs.gov:qdds2Mary:2222:2222:tigger@tsu.edu:From Reston to Mary
dmc.iris.washington.edu:qdds2Mary:2222:2222:tigger@tsu.edu:From Seattle to Mary
/pre>

In the above, two different passwords are used for the two hubs but they could be the same. /p>

Notice that each line is composed of tokens separated by the ":" character. The tokens are:

  1. The internet IP address to which to connect. It can be in text form or numeric form.
  2. A password for this connection. You create this. The password you create must be the same in your comm.lst and in the corresponding comm.lst at the other end of the connection.
  3. The UDP port at the internet address to communicate through
  4. The TCP port at the internet address to communicate through
  5. An e-mail address. For a hub, it should be the e-mail address of each leaf. For a leaf, it is the person running the leaf's e-mail address.
  6. A comment field.

QDDS.config

The QDDS.config sets parameters for your environment. Here is an example QDDS.config file with default parameters. That is, if a line is left out, that parameter will be set to the value given:


I AM A HUB: false
TRANSIENT LEAF: true
ALLOW TRANSIENT LEAVES: false
POLL DIRECTORY: polldir/
OUTPUT DIRECTORY: outputdir/
STORAGE DIRECTORY: storagedir/
LOG DIRECTORY: logdir/
TEMPORARY DIRECTORY: tempdir/
LISTEN PORT TCP: 2222
LISTEN PORT UDP: 2222
DAYS TO RETAIN OUTPUTDIR FILES: 2
DAYS TO RETAIN STORAGEDIR FILES: 2
DAYS TO RETAIN LOG FILES: 2
POLL WAIT TIME: 15
MINUTES ALIVE WAIT: 2
MINUTES TO CHECK TRANSIENTS: 20
MINUTES TO CHECK HUBS: 20
REQUEST ALL FILES: false
REQUEST ALL FILES MERGE: false
MAXIMUM RESENDS: 100
EXECUTE:
LOG LEVEL: 5
COMMLST FILE NAME: comm.lst
CURRENT FILE ID FILE NAME: curr_file_id
SAVE MAX RECEIVED FILE NAME: save_max_received
CONSOLE LOG LEVEL: 0
SMTP SERVER:  [smtp server]
EMAIL TO: [email address of administrator]
EMAIL FROM: [email address of administrator]

NOTE: In above, the administrator will be sending mail from and to him/herself. The only use of the e-mail notification at present is to send the administrator a message when QDDS is started. This only happens if SMTP SERVER, EMAIL TO, and EMAIL FROM have all been specified.

The following table documents the QDDS.config file entries:

Command Default value Comment
I AM A HUB: false Informs QDDS as to whether you are a hub or a leaf. If this line is omitted, a leaf is assumed.
TRANSIENT LEAF: true Only used if you are a transient leaf. A transient leaf is a leaf which is not registered with a hub. It registers online.
ALLOW TRANSIENT LEAVES: true Only used by a hub to allow or not allow transient leaves.
POLL DIRECTORY: polldir The directory into which you place an event file which is to be distributed to others
OUTPUT DIRECTORY: outputdir The directory into which QDDS places files which it receives through the network. It is possible to send files to more than one directory by including multiple OUTPUT DIRECTORY lines in QDDS.config.
STORAGE DIR: storagedir The directory which a hub places files which have been sent. These files are numbered event.1, event.2, etc. If a leaf misses a file, it requests it and the hub retrieves it from the storage directory.
LOG DIRECTORY: logdir Directory into which logs of QDDS activity are places. Each log file has a name of the form YYYYMMDDhhmm. A new file is created each midnight. If logging is turned off (See LOG LEVEL below), then these log files are not created.
TEMPORARY DIRECTORY: tempdir A directory into which output files are written and then "moved" to the final destination. This helps eliminate the problem of another process reading a partial file.
DAYS TO RETAIN OUTPUTDIR FILES: 2 How long to retain the files in the outputdir directory. This keeps the disk from filling up but still retains some files in case of a problem. If you set to zero, no files fill be erased.
DAYS TO RETAIN STORAGEDIR FILES: 2 How long to retain the files in the storagedir directory. If you set to zero, no files fill be erased.
DAYS TO RETAIN LOG FILES: 2 How long to retain the files in the log directory. If you set to zero, no files fill be erased.
POLL WAIT TIME: 2 The time, in seconds, for how often QDDS checks the polldir to see if a file is there.
CHECK MISSED WAIT: 2 The time, in minutes, to wait before requesting a missing file.
MINUTES ALIVE WAIT: 2 The number of minutes between alive messages being sent out. They are only sent by hubs and by transient leaves.
MINUTES TO CHECK TRANSIENTS: 20 Only for hubs. It specifies how often it checks to see if all the transient leaves are still "alive." If they are not, they are deleted from the active list of hosts.
MINUTES TO CHECK HUB: 20 Only for transient leaves. It specifies how often to see if the transient leaf's hubs are still alive.
MAXIMUM RESENDS: 100 The maximum number of events a leaf can request from a hub. This is set to some reasonable value such as 100 to prevent the case of a leaf being off-line for several days or weeks or even months and coming back up only to request thousands of events.
REQUEST ALL FILES: false If true, send all files in the storage director when started. One reason to use this setting is to allow the person starting a leaf to immediately see data coming.
REQUEST ALL FILES MERGE: false If true, send all files in the storage director when started. Send as one file. Receiving leaf unpacks into individual files.
EXECUTE:   Execute an external process. In a Unix environment, I have been able to make this work only if the external process is a script file with the first line being "#!/bin/csh". In a PC environment, be sure to specify the entire file name, e.g. "TEST.EXE" or "MINE.BAT". The external process is executed whenever a new event is added to the output directory.
LOG LEVEL: 0 Determines how much logging activity you wish. See below for values and their meaning.
COMMLST FILE NAME: comm.lst File name of file containing list of hubs. See above.
CURRENT FILE ID FILE NAME: curr_file_id File name of file which keeps track of this host's current file ID.
SAVE MAX RECEIVED FILE NAME: save_max_received File name of file which keeps track of the highest message number received from each host.
CONSOLE LOG LEVEL: 0 The CONSOLE LOG LEVEL allows you to set a log level for messages to the console so that a different level can be applied to the log file (LOG LEVEL) than to the console.
SMTP SERVER: _ SMTP server
EMAIL TO: _ Email address of administrator
 

LOG LEVELS AND THEIR MEANINGS

level What is logged
0 Nothing
1 Exceptions, changes to distribution list, console messages
2 Events
3 Distribution of events at host level
4 Polling
5 Everything else. This level is used mostly for debugging.

Transient Leaves

QDDS supports "permanent" leaves and "transient" leaves. A permanent leaf is one known to a hub through its entry in the hub's comm.lst file. Such a permanent leaf can stop running for any length of time with no problem. When it starts back up, it will again receive events. Any events lost during its down time will be requested when the leaf comes back up. A permanent leaf receives "alive" messages but it does not send alive messages. When it receives an alive message, it responds with an "Alive Reply": message which contains the Version and Level of the copy of QDDS it is running.

A transient leaf is not known to the hub when the hub starts. When the transient leaf starts, it issues a "request to register" command to the hub. When the hub accepts the request, it creates a new file called "comm.lst.trans" and writes the leaf's data into it. Any number of other transient leaves can be added in a similar manner. A transient leaf sends alive messages periodically to each hub to which it is attached. If a period of time goes by with no alive messages from a transient leaf, it is removed from the hub's in-memory list of leaves. The comm.lst.trans file is re-written with the leaf removed. The default time for checking on transient leaves is one hour but this can be set to any number of minutes in the QDDS.config file.

A transient leaf must have the user's e-mail address in the fifth field. This address is transmitted to the hub which stores it in its comm.lst.trans file and uses the address to send e-mail to all of its leaves using the SendMail program.

All Alive messages and Alive Reply messages contain the Version and Level of the copy of QDDS. This is in the format: Version 1.2 Level 2002.04.24.

The Alive and Reply messages from a leaf to a hub contain the current event ID number for that hub-leaf pair according to the leaf. The alive messages from a hub to a leaf contain the same information but according to the hub. In most cases these numbers will be the same. They will not be the same if the leaf or the has been offline and is catching up.

To start up a leaf as a transient leaf, create a comm.lst as above.

Also, add the following lines to your QDDS.config:

TRANSIENT LEAF: true
MINUTES ALIVE WAIT: 2 (or whatever)

Back to top

 

Messages

There are four types of messages:

  1. Newly created messages with no number assigned. This is a message sent by a leaf to its hubs when it finds an event in its polldir.
  2. A distributed message which has been assigned a number. This is a message sent by a hub to all of its leaves when it (1) finds an event in its polldir or (2) receives a new message from a leaf.
  3. Alive Message which are sent periodically from a hub to all of its leaves. The Alive message contains the Version and Level of the copy of QDDS. Also, transient leaves send alive messages to each hub. If a long period of time goes by without the hub receiving alive messages from a transient leaf, it discards that leaf from the comm.lst.tran file.
  4. Alive Reply messages which are sent from a permanent leaf to a hub after the leaf has received an Alive message from the hub. The Alive Reply message also contains the Version and Level of the copy of QDDS so that a hub can verify that its leaves are running up-to-date versions of QDDS.

If a hub receives a new message it assigns a number to it and stores it in the output directory and the storage directory. It then distributes it to all of its attached hosts.

Back to top

Fire walls

If you are behind a firewall, you may have to have your security folks "poke a hole" in the firewall to allow messages to go through. The default TCP and UDP ports are both 2222. On some fire walls this is called "port forwarding." Small messages (less than 1000 bytes) are sent via UDP but larger ones are sent via TCP. That is why you need both opened up. Most messages sent via QDDS are small and UDPs are much faster.

Running QDDS

QDDS is stored in a Java package called qdds. The package qdds is, in turn, stored in a QDDS.jar (Java ARchive) file. The routine which starts things up is the module QDDS in the package qdds. The process is started by executing java as follows:

java -jar QDDS.jar

Note: The above command line is the same regardless of the operating system.

After starting QDDS, you should see a number of messages and then the program should settle down with messages:

Files in polldir:

In order to see if things are working, copy one of the event files (event.1, event.2 etc.) into polldir. Within 15 seconds you should see messages telling you what QDDS is doing with the file.

Auxiliary Programs

Event.java

The Event.java program displays recent events received and placed in the output directory. It polls outputdir periodically and displays new events as they come in if the file is in the Cubic Format. NOTE: It then erases each file after displaying its content!

To use:

java Event

SendMail.java

The SendMail program sends an e-mail message to all users in a hub's comm.lst and comm.lst.trans files. The SendMail.class program is in the QDDS.jar file. To use:

java -cp QDDS.jar qdds.SendMail filename smtp_server from_email

where filename is the name of the file which contains the message to be sent and smtp_server is the name of the SMTP server

If you are on a Windows system:

As indicated above, run QDDS by typing:
java -jar QDDS.jar

Automatically Starting QDDS in Windows

If you want QDDS to start automatically when you start Windows,
you must add a command to the Start Up folder.

First create a STARTQDDS.BAT file in the root directory:

cd \qdds
java -jar qdds.QDDS
Then do the following:

If you are on a Unix System

It is a good idea to create a Unix script file to start
QDDS whenever the workstation is logged onto.
The script below checks to see if QDDS is already running
and if it is, it just exits.
You also should execute this script as a cron job so that
if QDDS stops for some reason, it will automatically be restarted.

Below is the script to start QDDS. You will have to modify it for your purposes:

#!/bin/sh

# a sh that runs $1 if $1 is not already running

# this ps statement will only return something if $1 is running
# otherwise the loop is skipped

PATH=$PATH:`dirname $0`
. HomeDir_info
#homedir=/home1/picker
PATH=:${homedir}/bin:/bin:/usr/ucb:/usr/bin
logfile=${homedir}/log

# this is a weird trick to get around the fact that ps will
# return a long string or nothing
# if it returns something the for loop runs and starter exits
# if it returns nothing the for loop is skipped and starter starts the program

for i in `ps -ef | grep "$1" | grep -v grep | grep -v starter`
do
        echo $0 says $1 is already running at `date` >> $logfile
        exit
done

echo $0 says $1 is not running at `date` >> $logfile
echo "$1 is not running; restarting" | mail your_pager_#@alphapage.airtouch.com
echo "$1 is not running; restarting" | mail your_email@host

# start things from the home directory and place them in background

cd $homedir/QDDS
java -jar QDDS.jar >& null &

The crontab is:

3 18 * * 1 /home1/picker/bin/newlog
02 * * * * /home1/picker/bin/starter_qdds "QDDS"
This will check every 2 minutes to see if QDDS is running and,
if not, will start it.

The first line will run the newlog script every Monday to compress the log files produced by the starter_qdds script.

The newlog script is:

#!/bin/sh

# keeps log files small, should be run once per week.
# Monday morning is a good time because it gives lots of
# time to notice problems

PATH=$PATH:`dirname $0`
. HomeDir_info
PATH=:${homedir}/bin:/usr/ucb:/bin:/usr/bin:/usr/earlybird/bin:/usr/picker/bin
logfiles="${homedir}/log"

# process the log files

for logfile in $logfiles
do
    # make sure files exist
    touch $logfile ${logfile}%.Z
    cp $logfile ${logfile}%
    cp /dev/null  $logfile
    echo new log created at `date` >> $logfile
    chmod a+w $logfile
    mv ${logfile}%.Z ${logfile}%%.Z
    compress -f ${logfile}%

done

Log Messages

There can be many types of messages in a log file:
Console messages:

How Events (files) are Sent

There are two ways a hub can obtain an event to distribute:
  1. By another application placing the file in the polldir. QDDS checks this directory every 20 seconds (or whatever time is specified in QDDS.config.)
  2. By a leaf sending a message to a hub. The leaf gets such a message in the same way the hub does, i.e. by a file being placed in its polldir.
Note: a leaf can be attached to more than one hub.
This could be done for redundancy or to receive events
from more than one network.


Back to top

Application Interface

An application can be a part of the QDDS system as (1) a provider of events or (2) as a recipient of events. To be a provider, the application must insert a file in the QDDS polldir. When QDDS finds a file, it reads it, dispatches it, and then erases the file. In a Unix system, files can be inserted into the polldir with the rcp command.

To be a user of events, the application can poll the QDDS outputdir and read any file which appears there. It then must erase the file.

Although the files can contain any data, at present the QDDS system is being used to distribute earthquake parametric data in the Cube format.

If you are interested in writing an application which processes the Cubic files, you might want to look at the Event.java example above.

Back to top

Callback Application

Eric Thomas has devised a callback method to receive messages directly instead of having them placed in a directory. When a callback method has been specified, the message goes to it rather than to the output directory. Here is an example of a callback method. It is in a Java program called ExtTest.java


//ExtTest.java:  Test program to exercise use of 'ExtWriteFileCallback'
//               object.
//
//  7/29/2002 -- [ET]
//

import qdds.Handler;
import qdds.ExtWriteFileCallback;
import qdds.QDDS;

/**
 * Class ExtTest is a test program to exercise the use of the
 * 'ExtWriteFileCallback' object.
 */
public class ExtTest
{
  public static void main(String [] args)
  {

    Handler.setExtWriteFileCallbackObj(new ExtWriteFileCallback()
        {
          public void writeFile(long msgNum, String msgStr)
          {
            System.out.println("*** Callback writeFile:  msg#=" +
                                msgNum + ", msgStr=\"" + msgStr + "\" ***");
          }
        });

    QDDS.main(args);
  }
}

ExtWriteFileCallback is a Java Interface defined in the file ExtWriteFileCallback.java:


//ExtWriteFileCallback.java:  Defines a callback method for capturing data
//                            that would have been written to files.
//
//  7/26/2002 -- [ET]
//

package qdds;

/**
 * Interface ExtWriteFileCallback defines a callback method for capturing
 * data that would have been written to files..
 */
public interface ExtWriteFileCallback
{
    /**
     * Callback method for capturing data that would have been written
     * to files.
     * @param msgNum message number.
     * @param msgStr message data string.
     */
  public void writeFile(long msgNum, String msgStr);
}


Both ExtTest.java and ExtWriteFileCallback.java are included in QDDS.jar. To use them, extract from the jar file:


jar xvf QDDS.jar *.java

If the name of your application is ExtTest.java, compile it and then execute it as:


java -cp QDDS.jar;. Exttest

Sample Application -- Seismic/Eruption

Seismic/Eruption is a Windows program which shows seismicity in speeded-up time. It has a database of earthquakes and volcanic eruptions from 1960 to near the present. Via QDDS, you can keep the database up-to-date to, depending on the quality of events sent, within a few minutes or a few hours. You must use a PC attached to the Internet running Windows 95, 98, NT, 2000, or XP.

Back to top

Appendix: Cube Format

The QDDS system uses formats from the CUBE system. 

Back to top