dasFlex server

Das servers typically provide data relevant to space plasma and magnetospheric physics research. To retrieve data, an HTTP GET request is posted to server by a client program. The server then replies with data formatted into the requested MIME-type. The most flexible reply is a dasStream version 3, though other output types are supported, such as CSV files.

This software, dasFlex provides middleware layer between server-side data readers, which stream data at full resolution to standard out, and remote client programs such as SDDAS, SPEDAS, and Autoplot, or custom programs written in Java (das2java), Python (das2py), IDL (das2pro, das2dlm ), or C (das2C ). Since dasFlex is a multi-MIME server, data may also be output as delimited text CSV and as CDF files, which are common in space-physics research.

When a request for data is received, dasFlex inspects the HTTP GET URL against local data source definitions and solves for the commands needed to produced the desired output. The command pipeline is then excuted and output is transmitted to the client as an HTTP body, or as part of a WebSocket communication session.

DasFlex itself is released under the GPL, but the core server merely acts as a command runner and output transport agent. Data producing program may be writen in any language, and released under almost any license.

Installation Prequisites

Compilation and installation of a dasFlex server has only been tested in Linux environments and depends on the following tools:

1. das2C >= 3.0
2. das2py >= 3.0
3. Python >= 3.7
4. Apache2, any remotely recent version

Automated Installation

For Conda Packages (uncommon) issue:

conda install -c dasdevelopers dasflex

Installation from Source

A summary is provided below. Additional instructions can be found in each individual package.

1. Get system software packages.

Rocky Linux Compatable

dnf install git gcc expat-devel fftw-devel openssl-devel python3-devel

Debian Linux Compatable

apt install gcc git libexpat-dev libfftw3-dev libssl-dev python3-dev

2. Get all sources in parallel directories

   mkdir -p git && cd git
   git clone https://github.com/das-developers/das2C.git
   git clone https://github.com/das-developers/das2py.git
   git clone https://github.com/das-developers/dasFlex.git

3. Build and test sources

   cd git/das2C
   make CDF=yes SPICE=yes
   make CDF=yes SPICE=yes test
   cd ../
    
   cd ../das2py
   make DAS2C=${PWD}/../das2C          # (optionally set PY_BIN first)
   make DAS2C=${PWD}/../das2C test
   make DAS2C=${PWD}/../das2C examples # (optional, long test)
   cd ../
   
   cd ../dasFlex
   make  # Automatically finds das2py in adjacent ../das2py
   make test

4. Install into self contained area

   export PREFIX=/var/www/dasflex  # Adjust to taste
   sudo mkdir -p $PREFIX
   
   cd git
   cd das2C
   make PREFIX=$PREFIX CDF=yes SPICE=yes install
   cd ../
   
   cd ../das2py
   python3 -m venv $PREFIX/venv
   $PREFIX/venv/bin/python -m pip install ./dist/*.whl
   cd ../
   
   cd ../dasFlex
   $PREFIX/venv/bin/python -m pip install ./dist/*.whl
   cd ../

Server Root Setup

To setup a server root area run dasflex_mkroot.

$PREFIX/venv/bin/dasflex_mkroot -h                  # View help text first

$PREFIX/venv/bin/dasflex_mkroot $PREFIX "My_Srv"    # Adjust ID as desired

$PREFIX/venv/bin/dasflex_mkroot -n $PREFIX "My_Srv" # (Alt) use -n to skip examples

The only file that the main CGI programs need to know about is dasflex.conf. This contains the locations of all other items such as the data source catalog. Feel free to break-up and move directories as desired. So long as dasflex.conf is updated with the new locations, top-level programs will be able to find all the necessary server components.

After the server root has been created, customize dasflex.conf. Starting with the SERVER_NAME, SERVER_ID settings.

You'll need a logging area that's writable by Apache.

sudo mkdir $PREFIX/log          # Also set 'LOG_PATH' in dasflex.conf
sudo chown apache $PREFIX/log   # Rocky
sudo chown www-data $PREFIX/log # Debian

Configure Apache - CGI

Apache configurations vary widely by Linux distribution and personal taste. The following procedure is provided as an example and has been tested on Rocky Linux 9. It assumes you're using a python virtual environment to host the main server scripts.

In the root of your SSL server, or in a <VirtualHost> section:

Alias /stream /var/www/dasflex/venv/bin/dasflex_cgimain
<Location /stream>

  # Provide CGI scripts with the location of the dasflex.conf file.
  # The location used here is just an example and changes depending
  # on the area selected by the `dasflex_mkroot` command above.
  SetEnv DASFLEX_CONFIG /var/www/dasflex/etc/dasflex.conf
  
  # Allow HTTP basic authentication to propogate to main script
  RewriteRule ^ - [E=HTTP_AUTHORIZATION:%{HTTP:Authorization}]

  Options ExecCGI FollowSymLinks
  SetHandler cgi-script
  RewriteEngine on
</Location>

Alias /log /var/www/dasflex/venv/bin/dasflex_cgilog
<Location /log>
  SetEnv DASFLEX_CONFIG /var/www/dasflex/etc/dasflex.conf
  Options ExecCGI FollowSymLinks
  SetHandler cgi-script
  RewriteRule ^ - [E=HTTP_AUTHORIZATION:%{HTTP:Authorization}]
  RewriteEngine on
  Require ip <LOG_IP_1> <LOG_IP_2>
</Location>

By default, authorization headers are not made available to CGI scripts. The re-write rule above allows the Authorization header to be passed down to the dasflex_cgimain script. This is needed to allow your server to support password protected data sources.

For security the log end-point shouldn't be open to the world, hence the <LOG_IP_1> <LOG_IP_2> replacement text above.

The main server script needs to know it's external URLs for generating links. If pick end-point URLs other than the default Alias directives above then update the following config entries in your dasflex.conf.

VIEW_LOG_URL = "log"     # Must be compatible with Alias directives,
MAIN_SRV_URL = "server"  # Full URLs are acceptable

Set the permissions of the log directory so that Apache can write logging information:

$ chmod 0777 /var/www/dasflex/log   # Or change the directory ownership

Finally, trigger a re-read of the Apache's configuration data:

$ sudo systemctl restart httpd.service
$ sudo systemctl status httpd.service

Test the server

Point your web browser at the path in the Alias statements above and see what you get. It should be a simple text message instructing you to setup you're dasflex.conf file if you haven't done so. By default these are:

http://localhost/das/server
http://localhost/das/log

If this works, try browsing your new server with Autoplot. To do so, copy the following URI into the Autoplot address bar and hit the green "Go" button:

vap+das2server:https://localhost/das/server

Next steps

The CGI scripts and worker programs read thier configuration data from the file:

/var/www/dasflex/etc/dasflex.conf

Take time to customize a few items in your config file such as the site_title and the contact_email. You may also want change the file ${PREFIX}/static/logo.png or even the style sheet at ${PREFIX}/static/dasflex.css to something a little nicer.

DasFlex is a caching and web-transport layer for data readers. Readers are the programs that generate the initial full resolution data streams. The entire purpose of dasFlex and das2 clients is to leverage the output of your reader programs to produce efficient, interactive science data displays. Example readers are included in the /var/www/dasflex/examples directory to assist you with the task of creating readers for your own data. These examples happen to be written in python, however there is no requirement to use python for your programs, in fact much more efficent compiled languages such as Java, D and C++ are more suitable for the task. Any language may be used so long as:

1. all data are written to standard output
2. all error messages are written to standard error

For further information on your dasFlex server instance, including:

• reader programs
• authentication
• request caching
• federated catalog integration

consult the wiki associated with this repository.

Examples License

All code in the examples directory (not including the temporary pycdf subdirectory) is release under the UNLICENSE and may be used without restrictions of any kind.

Experimental Feature: WebSocket support

The dasFlex server includes a websocket daemon for communicating with real-time data sources. This is not needed for standard server functionality, but it is very useful for supporting hardware development, since technicians want to see instrument data immediately. The assumption in the instructions below is that Apache will serve as the front door to the included dasflex_websocd daemon and will handle encryption/decription. The websocket daemon will only listen to the local host and all backend communication between Apache and dasflex_websocd will be unencrypted.

First add the trio-websocket package to your server virutal environment:

$ /var/www/dasflex/venv/bin/python -m pip install trio_websocket

Next, make sure mode the following apache modules are enabled:

a2enmod proxy proxy_wstunnel proxy_http rewrite

For RHEL-like systems, check the conf files in /etc/httpd/conf.modules.d.

Second in your applicable SSL server add the following. If you're using the default SSL server on Ubuntu the file is located at /etc/apache2/sites-enabled/default-ssl.conf or for RHEL, /etc/httpd/conf.d/ssl.conf.

<Location "/dasws/" >
  RewriteEngine on
  RewriteRule ^ - [E=HTTP_AUTHORIZATION:%{HTTP:Authorization}]
  ProxyPreserveHost on
  ProxyPass "ws://localhost:52242/dasws/"
  ProxyPassReverse "ws://localhost:52242/dasws/"
</Location>

Here the value ws://localhost:52242/dasws/ should be whatever you've specifed for the WEBSOCKET_URI in your dasflex.conf file.

A client program is included for testing your websocket server. An example of running it for the included spectra example would be:

$ python3 dasflex/test/ws_test_client.py \
   wss://localhost/dasws/examples/spectra/flexRT \
   read.time.min=1979-03-01T12:26:11 \
   read.time.max=1979-03-01T12:29:24 \
   format.serial=text

The websock server should run as the same user as the regular CGI server. This is critical because the log to the same files. That bears reapeating:

NOTE Run dasflex_websocd as the apache user on your system.

To do this the following command will work work on Debian and derivitives:

sudo su -s /usr/bin/bash -c "/path/to/dasflex_websocd 127.0.0.1 52242 -D /path/to/log/websock.pid" www-data

The user account on Rocky Linux is different, but otherwise the command is the same. A system-d unit file will be created as time permits.

To stop your server send SIGINT to the runing daemon

sudo kill -INT $(cat /path/to/log/websock.pid)