# README FOR READ ROUTE RECORD (triple R)
## ABOUT

RRR is a general purpose acquirement, transmission, processing and storage daemon for all kinds of
measurements or messages. RRR has a variety of different modules which can be chained together to
retrieve, transmit, modify and save messages.

RRR can read data of different types from external programs or sensors using UDP, TCP, UNIX sockets,
piping or files. It is also possible to run self made scripts inside of RRR to modify messages as
they pass through.

- Acquire messages using piping, UNIX sockets, UDP/TCP-packets
- Transfer messages using HTTP, MQTT, UDP or TCP
- Modify messages using Perl or Python
- Saving messages using InfluxDB and MySQL

Application examples may include:

- Barcode scanners
- Sensors (temperature, pressure, voltage etc.)
- PLCs
- Message forwarding, media or protocol conversion etc.

Please create an issue on the GitHub page https://github.com/atlesn/rrr if you are unable to use a
particular device with RRR. Please not that RRR is a general purpose program and does not support
any particular hardware unless configured to do so.

To send data to RRR, the data should be in some form of predictable binary or textual format. Data may
also be acquired directly by RRR if you write a custom Perl or Python script to acquire readings from
the source, for instance when the data source is a device on a computer.

RRR supports being run by SystemD and has native Debian/Ubuntu and ArchLinux packages. In addition, RRR
may be compiled and run on FreeBSD or other Linuxes including Raspbian. Other systems might work but
are not supported.

## LICENSES

RRR is released under the GPLv3 with some exceptions for self written modules. Please checkout
the `LICENSE.*`-files regarding this.

## DEVELOPMENT

It is possible to write custom modules in Perl, Python and C. All three have similar easy to use
interfaces to RRR. The man page for rrr.conf has more information on this, also check out `README.dev`.

If you are a developer and would like to see more functionallity or have some ideas, please consider
forking the GitHub repository.

## QUICK START

### DOWNLOAD

#### Compile from source

	$ git clone https://github.com/atlesn/rrr.git

See *COMPILE* section below.

#### Pre-compiled package Debian Buster amd64/i386 using APT

	$ sudo apt install curl
	$ curl -s https://apt.goliathdns.no/atle.gpg.key | sudo apt-key add -
	$ sudo sh -c "curl -s https://apt.goliathdns.no/goliathdns-buster.list > /etc/apt/sources.list.d/goliathdns.list"
	$ sudo apt update
	$ sudo apt install rrr
	
If your Debian system does not have *sudo* installed, remove *sudo* from all commands and run them as root.
	
#### Pre-compiled package for Ubuntu Focal amd64 using APT

	$ curl -s https://apt.goliathdns.no/atle.gpg.key | sudo apt-key add -
	$ sudo sh -c "curl -s https://apt.goliathdns.no/goliathdns-focal.list > /etc/apt/sources.list.d/goliathdns.list"
	$ sudo apt update
	$ sudo apt install rrr

#### Compile on ArchLinux using PKGBUILD file

	$ mkdir rrr
	$ cd rrr
	$ curl -s -o PKGBUILD https://raw.githubusercontent.com/atlesn/rrr/archlinux/misc/archlinux/PKGBUILD
	$ makepkg -cf
	$ sudo pacman -S perl
	$ sudo pacman -U rrr-*.zst

### COMPILE

Compiling the source requires some basic knowledge on how to build a program using Autotools. Please
look online if you have not done this before.

See `./configure --help` for flags to use for disabling modules with dependencies (perl, mysql etc.).

	$ autoreconf -i
	$ ./configure
	$ make
	$ sudo make install		-- Skip if you do not wish to install RRR on the filesystem

It is also possible, if you are on Ubuntu, Debian or similar, to build a `.deb` package. Look online
on detailed information about this and on which packages you need to build `.deb` packages.

	$ dpkg-buildpackage
	$ sudo dpkg -i ../rrr*.deb

### RUN

To start the program with modules, a config file must first be made. Write this into a file
called `rrr.conf`.

	[my_source]
	module=dummy
	dummy_no_generation=no
	
	[my_target]
	module=raw
	senders=my_source

Then, run this:

	$ rrr --debuglevel 2 rrr.conf

You can see that `my_source` generates messages which `my_target` then reads. Since RRR is designed to run forever,
use `Ctrl+C` to exit. RRR might produce some error messages when the different modules shut down, this is normal.

Keep reading below for more examples, and refer to `man rrr`, `man rrr_post` and `man rrr.conf` for more detailed information on how to configure `rrr`.

### RUN AND READ MANUALS WITHOUT INSTALLATION

	$ cd /directory/to/rrr/source/which/you/have/already/compiled
	$ man ./src/misc/man/rrr.conf.5
	$ man ./src/misc/man/rrr.1
	$ man ./src/misc/man/rrr_post.1
	$ ./src/rrr my_rrr_test_configuration.conf

## MODULES

Here's a short list of modules to start with, more are listed in the `rrr.conf` man page.

### ip (read and send data records using UDP and TCP)
This module handles inbound and outbound IP traffic. It can parse incoming data and organize it into RRR arrays,
and also send data to external hosts based on address information it the messages it receives from other modules.

The IP module is the binding link between RRR and external devices using IP networking.

### mqttbroker (run an MQTT broker)
Starts an MQTT broker which any MQTT client can use to exchange messages.

### mqttclient (run an MQTT client)
Starts an MQTT client which can connect to any MQTT server. Messages can be read from other modules and published
to a MQTT broker, and it is possible to subscribe to topics to receive messages other modules can receive. Two RRR MQTT
clients can exchange RRR messages through any MQTT broker, and an arbitary number of clients can run in each RRR program.

Below follows an example configuration which uses RRR to receive data records from an external MQTT client and save it to
an InfluxDB database:

	[my_mqtt_broker]
	module=mqttbroker
	
	[my_mqtt_client]
	module=mqttclient
	mqtt_server=localhost
	mqtt_subscribe_topics=a/+/#

	# Array definition of the data received from the MQTT broker
	mqtt_receive_array=fixp#loadavg,sep1,ustr#uptime,str#hostname
	
	[my_influxdb]
	module=influxdb

	# Read messages from the MQTT client
	senders=my_mqtt_client

	# Parameters used when writing to the InfluxDB server
	influxdb_server=localhost
	influxdb_database=mydb
	influxdb_table=stats

	# Tags and fields to retrieve from the received RRR messages and write to InfluxDB
	influxdb_tags=hostname
	influxdb_fields=uptime,loadavg->load

Use your faviourite MQTT client to publish a message with a data record to the RRR broker
which the RRR MQTT client then receives and parses. Refer to the `ARRAY DEFINITION` section
below on how to specifiy different data types.

	mosquitto_pub -t "a/b/c" -m "3.141592,12345678\"myserver.local\""
	
Please note that RRR does not provide a stand-alone MQTT client.

### ipclient (send messages between RRR instances over UDP)
This module is useful when messages need to be transferred over lossy connections where TCP doesn't work very well.

The `ipclient` module keeps track of all messages and makes sure that they are delivered excactly once (like MQTT QOS2).

The module may function both as a server which accepts connections and a client (also simultaneously).

An RRR native protocol is used to transfer messages which means that and RRR program only can communicate with other RRR
programs using this module.

### mysql (store messages to database)
Reads messages from other modules and stores them in a MySQL database.

### influxdb (store messages to database)
Reads messages from other modules and stores them in a an InfluxDB database.

### socket (read data records from a UNIX socket)
The socket module listens on a socket and parses data records it receives before forwarding them
as RRR messages to other modules. Expects to receive records with data types defined in an `array definition`. This
is useful when it's practical to pass over data to RRR locally using scripts.

A binary, `rrr_post`, is provided to communicate with the socket module. The type of input data is defined in it's
arguments, and an RRR array message for every record is created and sent in to RRR over the socket.

Below follows an example configuration where a data record is created locally and then sent to RRR using `rrr_post`
to be saved in an InfluxDB database.

	[my_socket]
	module=socket
	socket_path=/tmp/my_rrr_socket.sock
	socket_receive_rrr_message=yes

	[my_influxdb]
	module=influxdb

	# Read messages from the socket module
	senders=my_socket

	# Parameters used when writing to the InfluxDB server
	influxdb_server=localhost
	influxdb_database=mydb
	influxdb_table=stats

	# Tags and fields to retrieve from the received RRR messages and write to InfluxDB
	influxdb_tags=hostname
	influxdb_fields=uptime,loadavg->load

Use `rrr_post` to parse the input data record, which in this case is provided on standard input.

	echo "3.141592,12345678,\"myserver.local\"" | rrr_post /tmp/my_rrr_socket.sock -a fixp#loadavg,sep1,ustr#uptime,sep1,str#hostname -f - -c 1

### python3 (generate and/or modify messages by a custom python program)
The python3 module use a custom user-provided program to process and generate messages. Special RRR-
objects which resemble RRR internals are provided.

A custom python program might look like this, checkout `man rrr.conf` on how to make it run.

       from rrr_helper import *
       import time
       
       my_global_variable = ""

       def config(rrr_config: config):
            global my_global_variable

            # retrieve a custom setting from the configuration file. The get()
            # will update the "was-used" flag in the setting which stops a
            # warning from being printed.
            print ("Received configuration parameters")
            my_global_variable = config.get("my_global_variable")

            return True

       def process(socket: rrr_socket, message: rrr_message):
            # Return False if something is wrong
            if my_global_variable == "":
                 print("Error: configuration failure")
                 return False

            # modify the retrieved message as needed
            message.timestamp = message.timestamp + 1

            # queue the message to be sent back (optional) for python to give to readers
            socket.send(message)

            return True

       def source(socket: rrr_socket, message: rrr_message):
            # Set an array value in the template message
            my_array_value = rrr_array_value()
            my_array_value.set_tag("my_tag")
            my_array_value.set(0, "my_value")

            my_array = rrr_array()
            my_array.append(my_array_value)

            message.set_array(my_array)

            # queue the message to be sent back (optional) for python to give to readers
            socket.send(message)

            # sleep to limit output rate
            time.sleep(1)

            return True



The `rrr_helper` Python module is built into RRR and is only available when the script is called from this program.

### perl5 (generate and/or modify messages by a custom perl script)
The perl5 module makes it possible to process and generate messages in a custom 
written perl script. The first and only argument to the source- and generate-functions
is always a message. Modifications to the message in the script will be done to the
original message, hence there is no need to return the message.

A message can however be duplicated one or more times by calling it's send()-method.

Below follows an example perl script.

	#!/usr/bin/perl -w

	package main;

	use rrr::rrr_helper;
	use rrr::rrr_helper::rrr_message;
	use rrr::rrr_helper::rrr_settings;

	my $global_settings = undef;

	sub config {
		# Get the rrr_settings-object. Has get(key) and set(key,value) methods.
		my $settings = shift;

		# If needed, save the settings object
		$global_settings = $settings;

		# Custom settings from the configuration file must be read to avoid warning messages
		# print "my_custom_setting is: " . $settings->get("my_custom_setting") . "\n";

		# Set a custom setting
		$settings->set("my_new_setting", "5");

		return 1;
	}

	sub source {
		# Receive a newly generated template message
		my $message = shift;

		# Do some modifications
		$message->{'timestamp'} = $message->{'timestamp'} - $global_settings->get("my_custom_setting");

		print "source:  new timestamp of message is: " . $message->{'timestamp'} . "\n";

		# Return 1 for success and 0 for error
		return 1;
	}

	sub process {
		# Get a message from senders of the perl5 instance
		my $message = shift;

		# Do some modifications to the message
		$message->{'timestamp'} = $message->{'timestamp'} - $global_settings->get("my_custom_setting");

		print "process: new timestamp of message is: " . $message->{'timestamp'} . "\n";

		# This can be used to duplicate a message, no need if we are not duplicating
		# $message->send();

		# Return 1 for success and 0 for error
		return 1;
	}

### raw (drain output from other modules)
Read output from any module and delete it. Can also print out some information for each message it receives.

### dummy (generate dummy measurements)
Create dummy messages with current timestamp as value at some interval. 

## ARRAY DEFINITION
Many modules use array definitions to describe the data records they receive, as seen in the configuration examples above.

A definition is a comma separated list of different types. Some types need to have a specified length, and other types
figure out the length automatically. RRR will always make sure that it is possible to determin the length
of each record, this is necessary to be able to separate the records from each other upon receival. If you are unsure about
whether a definition is valid, just try to use it and RRR will give you an error message if it isn't.

Sequential fixed length values of the same type may be grouped together in a single items, and each item may be tagged to
ease the extraction of data in later processing.

The specification of an array definition is like this, and below follows more detailed explanations.

	type1[length1][s|u|][@count1][#tag1][,type2[length2][s|u|][@count2][#tag2]][,...]
	
- `type` - Identifier name of the type
- `length` - Length in bytes of the type (if required)
- `count` - Item count of the sepcific type, defaults to 1 if not specified
- `tag` - Optional custom identifier tag of the type

### FIXED LENGTH TYPES
These types require the `length` field to be specified.

- `be` - Unsigned number in big endian binary format. Length must be in the range 1-8.
- `le` - Unsigned number in little endian binary format. Length must be in the range 1-8.
- `h` - Unsigned number in the endianess of the machine. Might be unsafe for network transfer. Length must be in the range 1-8.
- `blob` - Arbitary binary data. Lengt must be in the range 1-1024.
- `sep` - One or more separator characters. Matches ! " # $ % & ' ( ) * + , - . / : ; < = > ? @ [ \ ] ^ _ ` { | } ~ LF CR TAB and SPACE. Length
must be in the range 1-64.

Types `be`, `le` and `h` may be followed by an `s` after the length specifier to indicate that the input number is signed. If instead
`u` or nothing is set here, the value is treated as unsigned. No other types may have sign flag set.

### WEAK DYNAMIC LENGTH TYPES
The length of these types are identified automatically and must not have length set. They
cannot be at the end of a definition, nor follow after each other.

- `ustr` - An unsigned integer encoded with ASCII characters 0-9. Stored with 64-bits.
- `istr` - A signed integer encoded with ASCII characters 0-9 optionally preceeded by - or +. Stored with 64-bits.
- `fixp` - The RRR fixed decimal type encoded with ASCII characters 0-9 (and A-F). May include a single dot . to separate integer from fraction,
and the integer part may be preceded with a sign (- or +). Stored with 64-bits where 1 bit is the sign, 39 bits is the integer and
24 bits are the fraction. May be preceeded with 10# or 16# to indicate use of base 10 or base 16 conversion, default is base 10.
- `nsep` - Will match number of any characters until a space character is found (LF, CR, TAB or SPACE).

### STRONG DYNAMIC LENGTH TYPES
The length of these types are identified automatically and must not have a length set. They may be at the end of a defintion.

- `msg` - A full RRR message complete with headers and checksums. A message has it's length inside the message header.
- `str` - An arbitary length string of characters beginning and ending with double quotes ". Double quotes inside the string must be escaped with \. The
surrounding quotes are not included in the final string.

## MESSAGES

A message contains some metadata and a class specifier which tells us what kind of
information and/or value it stores.
* Metadata for a message:
  * `Timestamp`
    Usually specified when a message was created.
  * `Class`
    The message class which describes what kind of information it stores:
    * `DATA`: Message with arbitary data
    * `ARRAY`: An array of data, for instance created by udpreader or socket
  * `Topic`
  	A topic to be used by MQTT modules
  * `Data`
    Data of the message (for ARRAY)

## CONTACT

github.com/atlesn/rrr

