Wednesday, April 20, 2022

Allowing LMIC to sleep soundly on a Feather bed

The previous two posts have provided most of what is required to allow a sketch to sleep between LMIC tasks.

The RTC is used to set an alarm to wake some number of seconds in the future, to go into standby mode, and to receive the alarm interrupt that brings the MCU out of standby. The fact it is being set to something like the current time is not relevant to these operations, but is useful if you wish to send your uplinks roughly aligned to certain times such as each hour, on the hour.

Then details were given on how to provide LMIC with a source of ticks that keeps running in standby mode, using the same oscillator the RTC uses, so there are no additional power requirements. Unfortunately that requires changes to one of the lowest levels of LMIC, making it incompatible with any other version and the Arduino library management system. The current structure of the Arduino port of LMIC makes it impossible to do this in a clean and compatible manner.

This post demonstrates how to use these features and modifications to go into standby mode between uplinks.

To begin with, if the sketch wishes to write messages to a serial port it must use Serial1, because Serial does not work after coming out of standby mode. So a USB-UART converter is required, attached to the RX/TX pins on the Feather. Serial1 does not need to reinitialised when waking from standby, it just keeps working.

Next, one more modification is required to LMIC. This new function allows the sketch to query when the next LMIC job is due to run.

Add the function declaration to oslmic.h, under the declaration for os_queryTimeCriticalJobs:

#ifndef os_getNextDeadline
ostime_t os_getNextDeadline(bit_t *valid);
#endif


And the function implementation goes in oslmic.c:

// Return the tick the next runnable or scheduled job is due.
//
// A runnable job want to run as soon as possible, so 'now' is returned
// if a runnable job exists. Otherwise, if there is a scheduled job the
// tick value for when that job will run is returned.
//
// If there is a runnable or scheduled job then valid will be non-zero
// upon return. If valid is zero, it means nothing is scheduled to run.
ostime_t os_getNextDeadline(bit_t *valid) {
if (OS.runnablejobs) {
*valid = 1;
return os_getTime();
}

if (OS.scheduledjobs) {
*valid = 1;
return OS.scheduledjobs->deadline;
}

*valid = 0;
return 0;
}


The sketch can now decide if it is reasonable to sleep, and if so for how long.

It seemed the safest algorithm to use was:

  • Do not sleep until the join is successful - ie busy-loop os_runloop_once() so LMIC has every chance to do the join.

  • Only consider sleeping between when one uplink/downlink completes and the next time sensor measurements and an uplink is due. This means go back to busy-looping os_runloop_once() after waking up on the assumption LMIC wants to do something soon.

  • Do not bother sleeping for less than a couple of seconds. Uncertainty in how the one-second-resolution time in the RTC lines up with the next job time means it is safer to ensure the sketch gives at least a one-second leeway.


Rather than picking out the many standby related parts of the sketch, here it is in total. It is a modification of the standard OTAA example.

#define serial Serial1

#include <lmic.h>
#include <hal/hal.h>
#include <RTCZero.h>

RTCZero rtc;

// This flag is used to avoid sleeping before the node had joined the
// network. Not very efficient but it ensures sleeping cannot get in
// the way of a join.
static bool joined = false;

// This flag is used to decide whether to make a network time request when
// an uplink is set. The network time request adds 1 byte to the payload.
// Once a network time response has been received this is set to true.
// It could be reset once a day or so to try and keep clock drift in check.
static bool timeOk = false;

// This flag is used to decide whether to see if there is time to sleep
// before the next LMIC job is due to run. Essentially, it is set to true
// when an uplink/downlink has completed because it is likely that sleeping
// is possible at that point, and then set to false when going to sleep on
// the assumption that when the node wakes up it should busy-loop
// os_run_once() because LMIC will be wanting to run a job soon.
static bool check_deadline = false;

// A buffer for printing log messages.
static constexpr int MAX_MSG = 256;
static char msg[MAX_MSG];

// A printf-like function to print log messages prefixed by the current
// LMIC tick value. Don't call it before os_init();
void log_msg(const char *fmt, ...) {
snprintf(msg, MAX_MSG, "% 012ld: ", os_getTime());
serial.write(msg, strlen(msg));
va_list args;
va_start(args, fmt);
vsnprintf(msg, MAX_MSG, fmt, args);
va_end(args);
serial.write(msg, strlen(msg));
serial.println();
}

static const uint8_t PROGMEM DEVEUI[8]={ ... };
static const uint8_t PROGMEM APPEUI[8]={ ... };
static const uint8_t PROGMEM APPKEY[16]={ ... };

void os_getArtEui (u1_t* buf) { memcpy_P(buf, APPEUI, 8);}

// This should also be in little endian format, see above.
void os_getDevEui (u1_t* buf) { memcpy_P(buf, DEVEUI, 8);}

// This key should be in big endian format (or, since it is not really a
// number but a block of memory, endianness does not really apply). In
// practice, a key taken from the TTN console can be copied as-is.
void os_getDevKey (u1_t* buf) { memcpy_P(buf, APPKEY, 16);}

static uint16_t counter = 0;
static osjob_t sendjob;

// Schedule TX every this many seconds (might become longer due to duty
// cycle limitations).
const unsigned TX_INTERVAL = 1800;

// Pin mapping for Adafruit Feather M0 LoRa, etc.
// /!\ By default Adafruit Feather M0's pin 6 and DIO1 are not connected.
// Please ensure they are connected.
const lmic_pinmap lmic_pins = {
.nss = 8,
.rxtx = LMIC_UNUSED_PIN,
.rst = 4,
.dio = {3, 6, LMIC_UNUSED_PIN},
.rxtx_rx_active = 0,
.rssi_cal = 8, // LBT cal for the Adafruit Feather M0 LoRa, in dB
.spi_freq = 8000000,
};

void onEvent (ev_t ev) {
switch(ev) {
case EV_JOINING:
log_msg("EV_JOINING");
break;
case EV_JOINED:
log_msg("EV_JOINED");

// Disable link check validation (automatically enabled
// during join, but because slow data rates change max TX
// size, we don't use it in this example.
LMIC_setLinkCheckMode(0);
joined = true;

// Send the first uplink immediately.
do_send(&sendjob);
break;
case EV_JOIN_FAILED:
log_msg("EV_JOIN_FAILED");
break;
case EV_TXCOMPLETE:
digitalWrite(LED_BUILTIN, LOW);
log_msg("EV_TXCOMPLETE (includes waiting for RX windows)");

// The uplink/downlink is done so now it is ok to check for
// a sleep window.
check_deadline = true;
break;
case EV_TXSTART:
digitalWrite(LED_BUILTIN, HIGH);
log_msg("EV_TXSTART");
break;
case EV_JOIN_TXCOMPLETE:
log_msg("EV_JOIN_TXCOMPLETE: no JoinAccept");
break;
}
}

void do_send(osjob_t* j) {
// Schedule next transmission so it is TX_INTERVAL from now, not from
// when this packet has finished sending.
os_setTimedCallback(&sendjob, os_getTime()+sec2osticks(TX_INTERVAL),
do_send);

// Check if there is not a current TX/RX job running
if (LMIC.opmode & OP_TXRXPEND) {
log_msg("OP_TXRXPEND, not sending");
} else {
// Keep asing for the time until the server provides it.
if ( ! timeOk) {
log_msg("Adding DeviceTimeReq MAC command to uplink.");
LMIC_requestNetworkTime(lmic_request_network_time_cb, 0);
}

// Prepare upstream data transmission at the next possible time.
counter++;
LMIC_setTxData2(1, (unsigned char *)&counter, sizeof(counter), 0);
log_msg("Packet queued");
}
}

void setup() {
pinMode(LED_BUILTIN, OUTPUT);
digitalWrite(LED_BUILTIN, LOW);

serial.begin(115200);

// LMIC init
os_init();
// Reset the MAC state. Session and pending data transfers will be
// discarded.
LMIC_reset();

LMIC_setLinkCheckMode(0);

// Reset the time in the RTC so it doesn't have random values before
// the network time has been received.
rtc.begin(true);

memset(msg, 0, sizeof(msg));

// Ask LMIC to start the join process.
LMIC_startJoining();
}

void printTime() {
log_msg("RTC time: %02d/%02d/%02d %02d:%02d:%02d",
rtc.getDay(), rtc.getMonth(), rtc.getYear(),
rtc.getHours(), rtc.getMinutes(), rtc.getSeconds());
}

static lmic_time_reference_t lmicTimeRef;

void lmic_request_network_time_cb(void * pUserData, int flagSuccess) {
if (flagSuccess != 0 && LMIC_getNetworkTimeReference(&lmicTimeRef)) {
timeOk = true;

// This is probably wrong, but it gets the node to within a couple
// of minutes. There must be a means of using tLocal and the current
// tick count to get a more accurate adjustment.

// GPS time is reported in seconds, and RTCZero also works with
// seconds in epoch values.
// 315964800 is the number of seconds between the UTC epoch and the
// GPS epoch, when GPS started.
// 18 is the GPS-UTC offset.
constexpr uint32_t adjustment = 315964800 - 18;
uint32_t ts = lmicTimeRef.tNetwork + adjustment;

// Adding rxDelay because the server sets the timestamp value to
// when it receives the uplink with the time request in it, but
// doesn't send the response until rxDelay seconds later.
ts = ts + LMIC.rxDelay;

// Add a second to cover RX1.
ts = ts + 1;

rtc.setEpoch(ts);

log_msg("tlocal = %ld, tNetwork = %lu, adjusted to UTS epoch = %lu",
lmicTimeRef.tLocal, lmicTimeRef.tNetwork, ts);
printTime();
}
}

static int32_t delta_osticks = -1;
static int32_t delta_seconds = -1;

void set_alarm() {
int32_t ss = (int32_t)rtc.getSeconds();
int32_t mm = (int32_t)rtc.getMinutes();
int32_t hh = (int32_t)rtc.getHours();

log_msg("Time now = %02d:%02d:%02d", hh, mm, ss);

int32_t delta = delta_seconds;
int32_t hh_delta = delta / 3600; delta -= (hh_delta * 3600);
// Will always be less than 1 hour.
int32_t mm_delta = delta / 60; delta -= (mm_delta * 60);
// Will always be less than 1 minute.
int32_t ss_delta = delta;

ss += ss_delta;
if (ss > 60) {
ss = ss % 60;
mm_delta++;
}

mm += mm_delta;
if (mm > 60) {
mm = mm % 60;
hh_delta++;
}

hh = (hh + hh_delta) % 24;

log_msg("Delta(s) = %d, wake at %02d:%02d:%02d",
delta_seconds, hh, mm, ss);

rtc.setAlarmTime((uint8_t)(hh & 0xff),
(uint8_t)(mm & 0xff),
(uint8_t)(ss & 0xff));
rtc.enableAlarm(RTCZero::MATCH_HHMMSS);
}

void loop() {
os_runloop_once();

// If LMIC hasn't joined yet, or is in a TX/RX state,
// let LMIC process and return without doing anything else.
if ( ! (joined || check_deadline)) {
return;
}

// This is used to decide whether to sleep so initialise it to
// a known value.
delta_seconds = -1;

// If an uplink/downlink has just finished, see if there is time to sleep
// before LMIC wants to do anything else.
if (check_deadline) {
check_deadline = false;
bit_t have_deadline = 0;
ostime_t timestamp = os_getTime();
ostime_t deadline = os_getNextDeadline(&have_deadline);

if (have_deadline) {
log_msg("Next deadline: %ld", deadline);
delta_osticks = deadline - timestamp;
delta_seconds = osticks2ms(delta_osticks)/1000;
log_msg("Delta from now: %ld, %ld s",
delta_osticks, delta_seconds);
} else {
// This likely means there is no user-level job scheduled
// and in this sketch that means no other job will ever be
// scheduled so this is the end of the line, but the sketch
// may as well keep busy-looping.
log_msg("No deadline");
}
}

// Given the RTC is used for the alarm and it has a 1 second granularity
// it seems prudent to provide a buffer for waking up in time to allow
// LMIC to run the next task.
if (delta_seconds > 2) {
delta_seconds--;

// Set the alarm in the RTC.
set_alarm();

// Ensure the log output is visible.
serial.flush();

// Don't try to sleep again after waking up, go back to busy-loop
// operation.
check_deadline = false;

// Go into standby mode until woken by an interrupt, eg the RTC alarm.
rtc.standbyMode();

// Disable the alarm in case it was set to some short interval and
// LMIC tasks will run for longer than that. It probably wouldn't
// cause trouble but may as well be sure.
rtc.disableAlarm();
}
}



Monday, April 18, 2022

More good times with LMIC and an AdaFruit LoRaWAN Feather M0

It is notoriously difficult to allow a node based upon the Arduino port of LMIC to sleep in a low power mode due to having to bring either LMIC or the Arduino core up-to-date with the time when the MCU awakes. It seems to be impossible without modifying either LMIC or the Arduino core.

This post covers the first half of the problem, how to give LMIC a time source that survives standby mode.

The basic problem seems to be that LMIC uses the Arduino micros() function to derive its internal tick value, and the value of micros() is not updated while the MCU is asleep. So if LMIC has a job scheduled, that job will not run at the appropriate time because LMIC (and the Arduino core) are now ‘in the past’ with respect to real-world time.

Various people have had a go at doing this by bring the time in the Arduino core up-to-date but at least in the AdaFruit SAMD21 M0 core I cannot see how to do this without changing the Arduino core because the varaibles are marked static, so cannot be referenced externally. The other thing I don’t like about this solution is that I don’t understand how LMIC derives its ticks from the micros() value, and if I must change something I may as well change code at the upper level (compared to the Ardiuino core code) to something I do understand.

Another problem with naive sleep schemes is that they may not handle cases where the LoRaWAN server causes a bit of a conversation to happen by downloading MAC commands, meaning the node should acknowledge those commands in a subsequent uplink that happens fairly quickly. I have observed up to to extra uplinks at a time caused by these interactions, with the uplinks happening in the space of a minute or two after the node’s own expected uplink. Simply going to sleep for some period of time after the expected uplink is done means these MAC command interchanges do not happen correctly and the server may continue to schedule downlinks to try and configure the node. Again, the next post will look at this in more detail.

Examing the LMIC os_ and hal_ layers of code it seems:

  • LMIC uses an unsigned 32-bit value for its internal ticks.

  • The frequency of the ticks is, by default, 32.768 KHz.

  • LMIC seems to be able to handle the roll-over of the 32-bit value even when it has jobs scheduled so far in the future that the value rolls-over many times.

Given the above, if a means to keep the ticks increasing at the fixed rate when the MCU is asleep can be arranged, LMIC should be able to start processing again when the MCU wakes up.

I took a look at the start-up state of the Feather when setup() is called to see how the oscillators and clock generators were configured, and what peripherals were free. That can be expanded upon in another post, but the good news is:

  • The external 32.768 KHz oscillator (XOSC32K) is enabled, and just needs the RUNSTDBY bit set to keep it going in standby mode to make it perfect for this job.

  • Clock generators 4 - 7 are all free to take a signal from an oscillator and send it to a peripheral.

  • The timer/counter peripherals are free, and can be put into a 32-bit counting mode.

  • The ultra-low-power internal 32.768 KHz oscillator could also be used, and always runs in standby mode, but it may not be accurate enough.

  • XOSC32K is used by the RTCZero library to drive the RTC, and I’m going to use the the RTC so I may as well use XOSC32K myself given it’s already running.

So hooking up XOSC32K to drive timer/counter pair 4/5 as a 32-bit counter, and have LMIC read the 32-bit COUNT register should satisfy the requirements above.

It seems like a cleaner solution to me, giving LMIC exactly what it wants with no bit shifting weirdness, and it just survives standby mode rather than having push time forward in the Arduino core, if you can even access the variables you need without modifying the core code.

The downsides are:

  • You must learn how to set these things up and it isn’t trivial; the datasheet is not easy to read and provides no code examples of how to do anything.

  • You must modify the LMIC hal_time_init() and hal_ticks() functions, meaning you lose compatibility with the standard MCCI Arduino LMIC library, and updating the library will lose your changes.

Here are the changes to hal.cpp in LMIC. Replace the hal_time_init() and hal_ticks() functions with these:

static void hal_time_init () {
    // Do not be tempted to split the register writes into separate statements
    // or use code that reads and writes them - the datasheet is quite specific
    // that some writes must just be whole-register writes or things don't work.

    // This is the same as how XOSC32K looks after the AdaFruit core has initialsed
    // the system, except the RUNSTDBY bit is being set so it keeps running during
    // standby mode.
    SYSCTRL->XOSC32K.reg = SYSCTRL_XOSC32K_RUNSTDBY
                         | SYSCTRL_XOSC32K_EN32K
                         | SYSCTRL_XOSC32K_XTALEN
                         | SYSCTRL_XOSC32K_STARTUP(6)
                         | SYSCTRL_XOSC32K_ENABLE;
    while (GCLK->STATUS.bit.SYNCBUSY);

    GCLK->GENCTRL.reg = GCLK_GENCTRL_ID(4)             // Specifies which generator is being configured
                      | GCLK_GENCTRL_GENEN             // Eenable the generator
                      | GCLK_GENCTRL_SRC_XOSC32K;      // Use XOSC32K as the source for the generator
    while (GCLK->STATUS.bit.SYNCBUSY);

    GCLK->CLKCTRL.reg = GCLK_CLKCTRL_GEN(4)            // Specifies which clock is being configured
                      | GCLK_CLKCTRL_CLKEN             // Enable the clock
                      | GCLK_CLKCTRL_ID(GCM_TC4_TC5);  // Feed the clock into peripheral timer/counter 4&5.
    while (GCLK->STATUS.bit.SYNCBUSY);

    // Disable timer/counter so it can be configured.
    while (TC4->COUNT32.STATUS.bit.SYNCBUSY);
    TC4->COUNT32.CTRLA.reg &= ~TC_CTRLA_ENABLE;
    while (TC4->COUNT32.STATUS.bit.SYNCBUSY);

    TC4->COUNT32.CTRLA.reg = TC_CTRLA_MODE_COUNT32     // Use 32-bit counting mode.
                           | TC_CTRLA_WAVEGEN(0)       // Use NFRQ mode so it just counts.
                           | TC_CTRLA_RUNSTDBY;        // Run when in standby mode.
    while (TC4->COUNT32.STATUS.bit.SYNCBUSY);

    // Enable the TC.
    TC4->COUNT32.CTRLA.reg |= TC_CTRLA_ENABLE;
    while (TC4->COUNT32.STATUS.bit.SYNCBUSY);

    // The 32-bit COUNT register of timer/counter 4&5 is now being incremented 32768 times a second.
}

u4_t hal_ticks () {
    // Signal we want to read the value of the COUNT register.
    TC4->COUNT32.READREQ.reg = TC_READREQ_RREQ | TC_COUNT32_COUNT_OFFSET;

    // Wait for the register value to be available.
    while (TC4->COUNT32.STATUS.bit.SYNCBUSY);

    // Read it.
    return TC4->COUNT32.COUNT.reg;
}

I had to add these #defines to lmic_project_config.h file to get everything to work, it’s too late in either oslmic.h or hal.h.

// LMIC requires ticks to be 15.5μs - 100 μs long
#define OSTICKS_PER_SEC 32768
// 1s/32768 = 0.00003052 so each tick is 30.52 μs, round up to 31 μs.
//
#define US_PER_OSTICK 31

With these changes LMIC has a very clean and simple source for its 32.769 KHz ticks. If you want to keep a 64-bit value for ticks, as LMIC may in future, it is a matter of enabling the overflow interrupt and incrementing the top-half of a uint64_t variable on each overflow, and putting the value of COUNT into the lower half whenever LMIC wants the value.

I have had a sketch running for at least 4 days with with these modifications, sending my own uplinks every 15 minutes, and it has been rock solid. I’ve used LMIC to schedule these 15 minute intervals, and been checking its idea of the time until the next job and it all seems correct. I may have missed the exact roll-overs, but looking back through the output I don’t see unexpected intervals.

Friday, April 15, 2022

Good times with LMIC and an AdaFruit LoRaWAN Feather M0

Until recently it has been a challenge to get anything like the current time on a Feather acting as a class A LoRaWAN node. The Feather has a built-in real time clock (RTC) but setting its value would require a downlink from your server-side application or an external module, in which case you didn’t need a RTC.

Recent updates to MCCIs version of LMIC and support in the community edition of the Things Stack make it easier to get close to the current time loaded into the Feathers RTC using the LMIC_requestNetworkTime function.

LMIC and the Things Stack now support v1.0.3 of the LoRaWAN specification, which added the ability to ask for the network time from a gateway using the DeviceTimeReq MAC command. If this command is included in an uplink, the gateway will schedule a downlink with the time in a DeviceTimeAns field of the MAC commands in the downlink. In my testing this downlink arrives in the RX windows of the uplink with the request - that is it arrives within 5 or 6 seconds. This might be different on networks with a shorter RX1 delay.

The DeviceTimeReq command is not supported in LoRaWAN versions less than 1.0.3, so your device must be registered on the Things Stack as a 1.0.3 device.

The time is set to the end of the transmission time of the uplink containing the DeviceTimeReq command, in GPS format which is seconds since 1980-01-06T00:00:00Z. To transform this to a UTC epoch-based value, you must:

  • Add the number of seconds since 1970-01-01T00:00:00Z to cover the basic difference between the two epochs.

  • There is something called the GPS-UTC offset, currently sitting at 18 seconds. I think you subtract this number from the value provided by the gateway.

  • The downlink is not received and processed until about RX1 delay + window seconds after the uplink was sent.

Given all the variables here, the time you end up with will be good, but not great. One advantage of that is that even if you have a site with many nodes, they won’t all be completely in sync so if you decide to uplink hourly, on the hour, the gateway still won’t get hit with a lot of uplinks at exactly the same time.

Based upon the usual LMIC demo sketches, here is an example.

 

#include <lmic.h>
#include <hal/hal.h>
#include <RTCZero.h>

RTCZero rtc;

static const uint8_t PROGMEM DEVEUI[8]={ ... };
static const uint8_t PROGMEM APPEUI[8]={ ... };
static const uint8_t PROGMEM APPKEY[16]={ ... };

void os_getArtEui (u1_t* buf) { memcpy_P(buf, APPEUI, 8);}

// This should also be in little endian format, see above.
void os_getDevEui (u1_t* buf) { memcpy_P(buf, DEVEUI, 8);}

// This key should be in big endian format (or, since it is not really a
// number but a block of memory, endianness does not really apply). In
// practice, a key taken from the TTN console can be copied as-is.
void os_getDevKey (u1_t* buf) {  memcpy_P(buf, APPKEY, 16);}

static uint16_t counter = 0;
static osjob_t sendjob;

// Schedule TX every this many seconds (might become longer due to duty
// cycle limitations).
const unsigned TX_INTERVAL = 120;

// Pin mapping for Adafruit Feather M0 LoRa, etc.
// /!\ By default Adafruit Feather M0's pin 6 and DIO1 are not connected.
// Please ensure they are connected.
const lmic_pinmap lmic_pins = {
    .nss = 8,
    .rxtx = LMIC_UNUSED_PIN,
    .rst = 4,
    .dio = {3, 6, LMIC_UNUSED_PIN},
    .rxtx_rx_active = 0,
    .rssi_cal = 8,              // LBT cal for the Adafruit Feather M0 LoRa, in dB
    .spi_freq = 8000000,
};

void printHex2(unsigned v) {
    v &= 0xff;
    if (v < 16)
        Serial.print('0');
    Serial.print(v, HEX);
}

void onEvent (ev_t ev) {
    switch(ev) {
        case EV_JOINING:
            Serial.println(F("EV_JOINING"));
            break;
        case EV_JOINED:
            Serial.println(F("EV_JOINED"));
            // Disable link check validation (automatically enabled
            // during join, but because slow data rates change max TX
            // size, we don't use it in this example.
            LMIC_setLinkCheckMode(0);
            break;
        case EV_JOIN_FAILED:
            Serial.println(F("EV_JOIN_FAILED"));
            break;
        case EV_TXCOMPLETE:
            digitalWrite(LED_BUILTIN, LOW);
            Serial.println(F("EV_TXCOMPLETE (includes waiting for RX windows)"));

            // Schedule next transmission
            os_setTimedCallback(&sendjob, os_getTime()+sec2osticks(TX_INTERVAL), do_send);
            break;
        case EV_RXCOMPLETE:
            digitalWrite(LED_BUILTIN, LOW);
            // data received in ping slot
            Serial.println(F("EV_RXCOMPLETE"));
            break;
        case EV_TXSTART:
            digitalWrite(LED_BUILTIN, HIGH);
            Serial.println(F("EV_TXSTART"));
            break;
        case EV_JOIN_TXCOMPLETE:
            Serial.println(F("EV_JOIN_TXCOMPLETE: no JoinAccept"));
            break;
    }
}


static bool timeOk = false;

void do_send(osjob_t* j){
    // Check if there is not a current TX/RX job running
    if (LMIC.opmode & OP_TXRXPEND) {
        Serial.println(F("OP_TXRXPEND, not sending"));
    } else {
        if ( ! timeOk) {
            Serial.println("Adding DeviceTimeReq MAC command to uplink.");
            LMIC_requestNetworkTime(lmic_request_network_time_cb, 0);
        }

        // Prepare upstream data transmission at the next possible time.
        counter++;
        LMIC_setTxData2(1, (unsigned char *)&counter, sizeof(counter), 0);
        Serial.println("Packet queued");
    }
    // Next TX is scheduled after TX_COMPLETE event.
}

void setup() {
    pinMode(LED_BUILTIN, OUTPUT);
    digitalWrite(LED_BUILTIN, LOW);

    delay(5000);
    while (! Serial)
        ;
    Serial.begin(115200);
    Serial.println("Starting");

    // LMIC init
    os_init();
    // Reset the MAC state. Session and pending data transfers will be discarded.
    LMIC_reset();

    LMIC_setLinkCheckMode(0);

    rtc.begin();

    // Start job (sending automatically starts OTAA too)
    do_send(&sendjob);
}


static lmic_time_reference_t lmicTimeRef;

void print2digits(int number)
{
    if (number < 10) {
        Serial.print("0");
    }
    Serial.print(number);
}

void printTime()
{
    // Print date...
    Serial.print(rtc.getDay());
    Serial.print("/");
    Serial.print(rtc.getMonth());
    Serial.print("/");
    Serial.print(rtc.getYear());
    Serial.print("   ");

    // ...and time
    print2digits(rtc.getHours());
    Serial.print(":");
    print2digits(rtc.getMinutes());
    Serial.print(":");
    print2digits(rtc.getSeconds());

    Serial.println();
}

void lmic_request_network_time_cb(void * pUserData, int flagSuccess)
{
    if (flagSuccess != 0 && LMIC_getNetworkTimeReference(&lmicTimeRef))
    {
        timeOk = true;

        // GPS time is reported in seconds, and RTCZero also works with seconds in epoch values.
        // 315964800 is the number of seconds between the UTC epoch and the GPS epoch, when GPS started.
        // 18 is the GPS-UTC offset.
        constexpr uint32_t adjustment = 315964800 - 18;
        uint32_t ts = lmicTimeRef.tNetwork + adjustment;

        // Adding rxDelay because the server sets the timestamp value to when it receives the uplink with the
        // time request in it, but doesn't send the response until rxDelay seconds later.
        ts = ts + LMIC.rxDelay;

        // Add a second to cover RX1.
        ts = ts + 1;

        rtc.setEpoch(ts);

        Serial.print("Network time request returned: tlocal = ");
        Serial.print(lmicTimeRef.tLocal);
        Serial.print(", tNetwork = ");
        Serial.println(lmicTimeRef.tNetwork);

        Serial.print("Adjusted to UTS epoch = ");
        Serial.println(ts);
        printTime();
    }
}


void loop() {
    os_runloop_once();
}

 

And the output:

 

Starting
Adding DeviceTimeReq MAC command to uplink.
Packet queued
EV_JOINING
EV_TXSTART
EV_JOINED
EV_TXSTART
Network time request returned: tlocal = 781908, tNetwork = 1334029344
Adjusted to UTS epoch = 1649994132
15/4/22   03:42:12
EV_TXCOMPLETE (includes waiting for RX windows)

Saturday, July 3, 2021

Creating pure JAX-RS web applications

How simple can a JAX-RS application be?

It is extremely simple to create a pure JAX-RS application. What's difficult is finding examples of how to do it.

There are no end of examples showing you how to create an application starting with a Maven archetype or an IDE and a JAX-RS implementation. But what you'll end up with is an enormous amount of cruft in your project directory tree and in your war file. You'll probably also end up with non-standard code using the 'helpful' extensions added to whatever implementation the tutorial author used, because they based their tutorial on one written by the JAX-RS implementation authors, who want to showcase their extensions to the standard.

This is not that. This tutorial shows how small a JAX-RS application can be, and how you don't need to download the universe to build and package it.

So what do you need?

  • A JDK. Java8+.
  • A text editor.
  • The JAX-RS api jar.
  • curl for downloading software and testing.

Assumptions

It is assumed you have a JDK installed and in your path and that JAVA_HOME points to it, and you have curl downloaded and in your path. You'd be a masochist to not use an IDE. But you don't need that for packaging the application.

Other than editing the source code this will be demonstrated entirely with the Windows 10 command shell. It would be easier with bash.

Project directory layout

This is very simple. A src directory for source files, a lib directory for the jax-rs-api jar, and war directory to build the war file from.


C:\Users\dajta\src\testwebapp>mkdir src\net\gweeps\webapp\model
C:\Users\dajta\src\testwebapp>mkdir src\net\gweeps\webapp\resources
C:\Users\dajta\src\testwebapp>mkdir lib
C:\Users\dajta\src\testwebapp>mkdir war\WEB-INF\classes

Download the JAX-RS api jar. I googled "jax-rs 2.1.1 api jar" to find it.

C:\Users\dajta\src\testwebapp>curl -s https://repo1.maven.org/maven2/javax/ws/rs/javax.ws.rs-api/2.1.1/javax.ws.rs-api-2.1.1.jar --output lib\javax.ws.rs-api-2.1.1.jar

Now the source files. There is an Application subclass to signal the war file contains a JAX-RS webapp, a test resource that provides a URI for requests, and a POJO representing some type of object in the application's data model.

src\net\gweeps\webapp\TestApplication.java


package net.gweeps.webapp;

// Nothing here about a JAX-RS implementation - this is pure JAX-RS API.
import javax.ws.rs.ApplicationPath;
import javax.ws.rs.core.Application;

/**
 * This class tells the runtime environment to look for
 * JAX-RS resources in the war file.
 *
 * <p>The fact that none of the superclass methods are over-ridden
 * causes the runtime environment scan for the resources.</p>
 *
 * <p>The specification says this is optional but recommended
 * behaviour for the runtime.</p>
 */
@ApplicationPath("api")
public class TestApplication extends Application {
}

src\net\gweeps\webapp\resources\TestResource.java


package net.gweeps.webapp.resources;

import java.util.logging.Logger;

// Nothing here about a JAX-RS implementation - this is pure JAX-RS API.
import javax.ws.rs.GET;
import javax.ws.rs.Path;
import javax.ws.rs.Produces;
import javax.ws.rs.QueryParam;
import javax.ws.rs.core.MediaType;

// The model POJO
import net.gweeps.webapp.model.Test;

/**
 * The main entry point for the test resource.
 *
 * <p>This is available at /testapp/api/test.</p>
 */
@Path("test")
public class TestResource {

    private static final Logger logger = Logger.getLogger(TestResource.class.getName());

    /**
     * An example sub-resource method.
     *
     * <p>The result is an instance of the POJO Test, encoded in JSON. This is to show how
     * JAX-RX just knows how to do this without and great amount of coniguration.</p>
     *
     * @param id a request id.
     * @return a {@link Test} object representing the model's recommendations, encoded in JSON.
     */
    @GET
    @Produces(MediaType.APPLICATION_JSON)
    public Test get(@QueryParam("id") String id) {
        logger.info(String.format("Received request id: %s", id));

        Test response = new Test(id, "A test object.");

        logger.info(String.format("Reponse for request id: %d, %s", id, response.toString()));
        return response;
    }
}

src\net\gweeps\webapp\model\Test.java


package net.gweeps.webapp.model;

import java.util.ArrayList;
import java.util.List;

/**
 * A POJO representing some object in the application model.
 *
 * <p>Note there are no annotations regarding JSON here.</p>
 */
public class Test {
	private String name;
	private String description;
	private List<String> details = new ArrayList<>();

	public Test(String name, String description) {
		this.name = name;
		this.description = description;
		details.add("Detail 1.");
		details.add("Detail 2.");
	}

	public String getName() {
		return name;
	}

	public void setName(String name) {
		this.name = name;
	}

	public String getDescription() {
		return description;
	}

	public void setDescription(String description) {
		this.description = description;
	}

	public List<String> getDetails() {
		return details;
	}
}

Compilation and packaging

To compile and package the application requires using javac and jar.

The java compiler will not compile everything under a particular directory, it wants a list of source files. The easiest way to do this in the Windows 10 command prompt is to use the dir command and the @ argument for javac:


C:\Users\dajta\src\testwebapp>dir /s /B *.java >sourcefiles.txt
C:\Users\dajta\src\testwebapp>javac -cp lib\javax.ws.rs-api-2.1.1.jar -sourcepath src -d war\WEB-INF\classes @sourcefiles.txt
C:\Users\dajta\src\testwebapp>cd war
C:\Users\dajta\src\testwebapp\war>jar cMf ..\testapp.war *
C:\Users\dajta\src\testwebapp>cd ..
C:\Users\dajta\src\testwebapp>jar tf testapp.war
WEB-INF/
WEB-INF/classes/
WEB-INF/classes/net/
WEB-INF/classes/net/gweeps/
WEB-INF/classes/net/gweeps/webapp/
WEB-INF/classes/net/gweeps/webapp/model/
WEB-INF/classes/net/gweeps/webapp/model/Test.class
WEB-INF/classes/net/gweeps/webapp/resources/
WEB-INF/classes/net/gweeps/webapp/resources/TestResource.class
WEB-INF/classes/net/gweeps/webapp/TestApplication.class

No XML files to be seen, or even a manifest.

Running the application

A simple way to run the application is to download either TomEE or Wildfly, copy the war file to the appropriate directory, and start the server.

TomEE


C:\Users\dajta\src\testwebapp>curl -O https://downloads.apache.org/tomee/tomee-8.0.6/apache-tomee-8.0.6-webprofile.zip

You must use version 8 of TomEE so you get a Jakarta EE 8 implementation. Jakarta EE 9 introduces breaking changes in the JAX-RS API package names. Awesome.

Unzip TomEE in place, and copy the war file into it:


C:\Users\dajta\src\testwebapp>copy testapp.war apache-tomee-webprofile-8.0.6\webapps

Run TomEE and the application will be deployed:


03-Jul-2021 13:44:58.844 INFO [main] org.apache.openejb.config.AppInfoBuilder.build Enterprise application "C:\Users\dajta\src\testwebapp\apache-tomee-webprofile-8.0.6\webapps\testapp" loaded.
03-Jul-2021 13:44:58.848 INFO [main] org.apache.openejb.assembler.classic.Assembler.createApplication Assembling app: C:\Users\dajta\src\testwebapp\apache-tomee-webprofile-8.0.6\webapps\testapp
03-Jul-2021 13:44:59.021 INFO [main] org.apache.openejb.assembler.classic.Assembler.createApplication Deployed Application(path=C:\Users\dajta\src\testwebapp\apache-tomee-webprofile-8.0.6\webapps\testapp)
...
03-Jul-2021 13:44:59.328 INFO [main] org.apache.openejb.server.cxf.rs.CxfRsHttpListener.logEndpoints REST Application: http://localhost:8080/testapp/api      -> net.gweeps.webapp.TestApplication@12cd9150
03-Jul-2021 13:44:59.331 INFO [main] org.apache.openejb.server.cxf.rs.CxfRsHttpListener.logEndpoints      Service URI: http://localhost:8080/testapp/api/test -> Pojo net.gweeps.webapp.resources.TestResource
03-Jul-2021 13:44:59.331 INFO [main] org.apache.openejb.server.cxf.rs.CxfRsHttpListener.logEndpoints               GET http://localhost:8080/testapp/api/test ->      Testget(String)
03-Jul-2021 13:44:59.348 INFO [main] sun.reflect.DelegatingMethodAccessorImpl.invoke Deployment of web application archive [C:\Users\dajta\src\testwebapp\apache-tomee-webprofile-8.0.6\webapps\testapp.war] has finished in [699] ms

WidlFly

I used WidlFly 23, the latest release at the time of writing. It still has Jakarta EE 8, rather than Jarkarta EE 9.


C:\Users\dajta\src\testwebapp>curl -O https://download.jboss.org/wildfly/23.0.2.Final/wildfly-23.0.2.Final.zip    

Unzip WidlFly in place, and copy the war file into it:


C:\Users\dajta\src\testwebapp>copy testapp.war wildfly-23.0.2.Final\standalone\deployments

Run WidlFly and the application will be deployed:


16:57:26,851 INFO  [org.jboss.as.server.deployment] (MSC service thread 1-7) WFLYSRV0027: Starting deployment of "testapp.war" (runtime-name: "testapp.war")
...
16:57:28,212 INFO  [org.jboss.resteasy.resteasy_jaxrs.i18n] (ServerService Thread Pool -- 79) RESTEASY002225: Deploying javax.ws.rs.core.Application: class net.gweeps.webapp.TestApplication
...
16:57:28,276 INFO  [org.wildfly.extension.undertow] (ServerService Thread Pool -- 79) WFLYUT0021: Registered web context: '/testapp' for server 'default-server'
16:57:28,320 INFO  [org.jboss.as.server] (ServerService Thread Pool -- 45) WFLYSRV0010: Deployed "testapp.war" (runtime-name : "testapp.war")

Testing

Use curl to connect to it and marvel at the fact the POJO was converted to JSON with no more effort than marking the sub-resource method with an annotation.


C:\Users\dajta\src\testwebapp>curl localhost:8080/testapp/api/test
{"description":"A test object.","details":["Detail 1.","Detail 2."]}

Monday, June 22, 2015

The BQ4845 RTC

The simplest external device I could see for my first attempt at a peripheral was the TI BQ4845 RTC. It's a DIP package and works like an SRAM, can have a 32,768Hz crystal attached directly to it, and has a simple interface for connecting and programming.

Monday, June 15, 2015

An expansion header board for the CPUville kit

With the C compiler working it was time to get back to the hardware and make it possible to experiment with new devices.

Saturday, May 30, 2015

stdio functions via the CPUville monitor

This post covers replacing the standard crt0.rel file and making stdio work with the CPUville monitor.