Wrong time data from NTPClient

[afrixx]

Describe the problem

Normally, this library provides correct time data, with a maximum 2s deviation.
But some days, let's say every 30 days, I get wrong time data from the library. Sometimes 5 minutes difference, sometimes 30 minutes. And although I have configured a refresh every 10 minutes, the time remains wrong for hours, sometimes only until a hard reset.

You could say it's a network problem, but I can access the webserver on the ESP all the time.

Additional context

I am using a Wemos D1 Mini board (ESP8266-based).

[WhymustIhaveaname]

It is an intricate problem. Can I have a look at your codes? You can include files in reply.

[afrixx]

Thanks @WhymustIhaveaname for having a look at my code.
I included the 2 basic files:

wortuhr.ino

#include <ESP8266WiFi.h>  //ESP8266 Core WiFi Library
extern "C" {
#include "user_interface.h"  //to set hostname
}

#include <DNSServer.h>         //Local DNS Server used for redirecting all requests to the configuration portal
#include <ESP8266WebServer.h>  //Local WebServer used to serve the configuration portal
#include <WiFiManager.h>       //https://github.com/tzapu/WiFiManager WiFi Configuration Magic

#include <NTPClient.h>          //NTP Client to get current UTC time
#include <Adafruit_NeoPixel.h>  //Adafruit NeoPixel library to control WS2812B stripes

#include <WiFiUdp.h>

#include <EEPROM.h>  // to store brightness and color

#include <ESP8266mDNS.h>  // Include the mDNS library

#include "clockwords.h"
#include "colors.h"

#define RGBW 1  // 0=RGB stripe, 1=RGBW stripe

// Which pin on the Arduino is connected to the NeoPixels?
// On a Trinket or Gemma we suggest changing this to 1
#define PIN D1

// How many NeoPixels are attached to the Arduino?
#define NUMPIXELS 114

// When we setup the NeoPixel library, we tell it how many pixels, and which pin to use to send signals.
// Note that for older NeoPixel strips you might need to change the third parameter--see the strandtest
// example for more information on possible values.
#if RGBW == 0
Adafruit_NeoPixel pixels = Adafruit_NeoPixel(NUMPIXELS, PIN, NEO_GRB + NEO_KHZ800);
#else
Adafruit_NeoPixel pixels = Adafruit_NeoPixel(NUMPIXELS, PIN, NEO_GRBW + NEO_KHZ800);
#endif


char wiFiHostname[] = "wortuhr";
long currentTimeZone = 3600;  //offset in seconds

////////////////////////////////
/// DO NOT MODIFY AFTER HERE ///
////////////////////////////////

WiFiManager wifiManager;

WiFiUDP ntpUDP;


unsigned long timeZoneValidUntil = 0;
unsigned long currentUtcEpochTime = 0;
unsigned long lastUtcEpochTime = 0;  //added to detect bigger time jumps >3h into the past
unsigned long lastTimeUpdate = 0;    //time when last time NTP client synced from server

// You can specify the time server pool and the offset (in seconds, can be
// changed later with setTimeOffset() ). Additionaly you can specify the
// update interval (in milliseconds, can be changed using setUpdateInterval() ).
NTPClient timeClient(ntpUDP, "de.pool.ntp.org", currentTimeZone, 600000);

int uint8_tSize = sizeof(uint8_t);

uint8_t color_red = 0;
uint8_t color_green = 0;
uint8_t color_blue = 0;
uint8_t color_white = 0;

uint8_t colorNr = 0;
uint8_t brightness = 0;

uint8_t languageMode = 0;
uint8_t rgbMode = 0;

int currentMinute = -1;
int currentSecond = -1;
bool doTimeRepaint = false;

bool secondsModeOn = false;
uint8_t specialMode = 0;
unsigned long secondsOnUntil = 0;
ESP8266WebServer webServer;


void setup() {
  Serial.begin(115200);
  Serial.println("");

  // Read EEPROM and load values
  InitEeprom();
  ReadEepromValues();
#if RGBW == 0
  Serial.println("*Wortuhr: Starting with RGB mode");
#else
  Serial.println("*Wortuhr: Starting with RGBW mode");
#endif

  pixels.begin();
  pixels.clear();
  Serial.println("*Wortuhr: Display FUNK");
  TurnByteArrayOn(&_word_funk[0], 255, 0, 0, 0);
  pixels.show();

  Serial.println("*Wortuhr: Connect to WIFI...");

  wifi_station_set_hostname(wiFiHostname);
  wifiManager.setAPCallback(configModeCallback);
  wifiManager.autoConnect("wortuhr");
  Serial.println("*Wortuhr: WIFI connected!");

  timeClient.begin();
  Serial.println("");

  // Start mDNS
  if (!MDNS.begin("wortuhr")) {  // Start the mDNS responder for esp8266.local
    Serial.println("*Wortuhr: Error setting up MDNS responder!");
  }
  Serial.println("*Wortuhr: mDNS responder started! You can connect by http://wortuhr.local\n");

  StartWebserver();
  StartApi();

  UdpInit();
}

void loop() {
  bool timeUpdated = timeClient.update();
  if (timeUpdated == true) {
    lastTimeUpdate = timeClient.getEpochTime() - currentTimeZone;
  }

  lastUtcEpochTime = currentUtcEpochTime;  //backup last time
  currentUtcEpochTime = timeClient.getEpochTime() - currentTimeZone;

  //check for time jumps backwards
  if (currentUtcEpochTime + (3 * 3600) < lastUtcEpochTime) {
    //time jumped back for >3h --> clear timeZoneValidUntil
    timeZoneValidUntil = 0;
    Serial.println("*Wortuhr: Time jumped back more then 3 hours, requesting new time zone information");
  }

  //check if time zone is not valid anymore
  if (currentUtcEpochTime > timeZoneValidUntil) {
    timeZoneValidUntil = currentUtcEpochTime + 60;  //security mechanism, to only ask the server every 60s at maximum (if not reachable etc.)
    SetTimeZone(currentUtcEpochTime);               //load and set the current time zone
  }

  //check if time is valid (> 1.1.1970)
  if (currentUtcEpochTime > (24 * 3600)) {
    int hour = timeClient.getHours();
    int minute = timeClient.getMinutes();  // Get the Minutes here and not the Seconds

    if (secondsModeOn == false && specialMode == 0) {
      if (minute != currentMinute || doTimeRepaint == true)  //time has changed!
      {
        doTimeRepaint = false;
        pixels.clear();  // Clear Stripe
        Serial.println("*Wortuhr: Display Time: " + timeClient.getFormattedTime() + ", utc epoch: " + String(currentUtcEpochTime));
        // Update Time
        UpdateTime(hour, minute);
        currentMinute = minute;
      }
    } else if (specialMode > 0) {
      if (specialMode == 1) { showXmasTree(); }
      if (specialMode == 2) { showEasterEggs(); }
      Serial.println("*Wortuhr: Display Time (SPECIAL MODE): " + timeClient.getFormattedTime());
      currentMinute = -1;

      if (secondsOnUntil > 0 && secondsOnUntil < currentUtcEpochTime) {
        secondsOnUntil = 0;
        specialMode = 0;
      }
    } else {
      int second = timeClient.getSeconds();
      if (second != currentSecond) {
        TurnSecondsOn(second);
        Serial.println("*Wortuhr: Display Time (SECONDS MODE): " + timeClient.getFormattedTime());
        currentMinute = -1;
        currentSecond = second;

        if (secondsOnUntil > 0 && secondsOnUntil < currentUtcEpochTime) {
          secondsOnUntil = 0;
          secondsModeOn = false;
        }
      }
    }
  } else {
    pixels.clear();  // Clear Stripe
    Serial.println("*Wortuhr: Could not get NTP time!");
    TurnByteArrayOn(&_word_ntp[0], 0, 0, 255, 0);  //display NTP
  }

  // Update Stripe
  pixels.show();

  //wait 1s and handle web client requests if necessary
  //wait 200ms when seconds are being displayed
  uint8_t count_max = (secondsModeOn == false && specialMode == 0) ? 10 : 2;

  for (uint8_t i = 0; i < count_max; i++)  //wait 1s
  {
    delay(100);
    webServer.handleClient();
  }

  UdpCheck();  //check for incoming UDP packages
}

void configModeCallback(WiFiManager *myWiFiManager) {
  //Wifi is now in access point config mode
  Serial.println("*Wortuhr: Entered access point config mode");
  TurnByteArrayOn(&_word_an1[0], 100, 100, 100, 100);
  TurnByteArrayOn(&_word_an2[0], 100, 100, 100, 100);
  pixels.show();
}

timezoneHelper.ino

#include <ArduinoJson.h>

WiFiClient wifiClient;

void SetTimeZone(unsigned long currentUtcEpochTime) {
  const char* timeZoneServer = "api.timezonedb.com";
  String timeZoneUrl = "/v2/get-time-zone?key=XXXXXXXXX&format=json&by=zone&zone=Europe/Berlin";

  const unsigned long HTTP_TIMEOUT = 10000;  // max respone time from server
  const size_t MAX_CONTENT_SIZE = 2048;      // max size of the HTTP response

  const unsigned long START_YEAR_2036 = 2082754800;

  //check if time is valid (> 1.1.1970) and before 1.1.2036 (NTP Bug of 7.2.2036)
  if ((currentUtcEpochTime > (24 * 3600)) && (currentUtcEpochTime < START_YEAR_2036)) {
    timeZoneUrl += "&time=";
    timeZoneUrl += String(currentUtcEpochTime);
  }

  if (!wifiClient.connect(timeZoneServer, 80)) {
    Serial.println("*Wortuhr: Could not connect to time zone server");
    wifiClient.stop();
    return;
  }

  wifiClient.print("GET ");
  wifiClient.print(timeZoneUrl);
  wifiClient.println(" HTTP/1.0");
  wifiClient.print("Host: ");
  wifiClient.println(timeZoneServer);
  wifiClient.println("Connection: close");
  wifiClient.println();

  // Skip HTTP headers so that we are at the beginning of the response's body
  // HTTP headers end with an empty line
  char endOfHeaders[] = "\r\n\r\n";

  wifiClient.setTimeout(HTTP_TIMEOUT);
  bool ok = wifiClient.find(endOfHeaders);

  if (!ok) {
    Serial.println("*Wortuhr: No response or invalid response from Timezone server!");
    wifiClient.stop();
    return;
  }

  //Parse JSON response
  const size_t bufferSize = JSON_OBJECT_SIZE(13) + 260;
  DynamicJsonBuffer jsonBuffer(bufferSize);

  //const char* json = "{\"status\":\"OK\",\"message\":\"\",\"countryCode\":\"DE\",\"countryName\":\"Germany\",\"zoneName\":\"Europe/Berlin\",\"abbreviation\":\"CEST\",\"gmtOffset\":7200,\"dst\":\"1\",\"dstStart\":1521939600,\"dstEnd\":1540688399,\"nextAbbreviation\":\"CET\",\"timestamp\":1526505696,\"formatted\":\"2018-05-16 21:21:36\"}";

  JsonObject& root = jsonBuffer.parseObject(wifiClient);

  const char* status = root["status"];  // "OK"
  //const char* message = root["message"]; // ""
  //const char* countryCode = root["countryCode"]; // "DE"
  //const char* countryName = root["countryName"]; // "Germany"
  //const char* zoneName = root["zoneName"]; // "Europe/Berlin"
  //const char* abbreviation = root["abbreviation"]; // "CEST"
  long gmtOffset = root["gmtOffset"];  // 7200
  //const char* dst = root["dst"]; // "1"
  //long dstStart = root["dstStart"]; // 1521939600
  long dstEnd = root["dstEnd"];  // 1540688399
  //const char* nextAbbreviation = root["nextAbbreviation"]; // "CET"
  //long timestamp = root["timestamp"]; // 1526505696
  //const char* formatted = root["formatted"]; // "2018-05-16 21:21:36"

  if (String(status) == "OK") {
    currentTimeZone = gmtOffset;
    if (dstEnd > 0) timeZoneValidUntil = dstEnd;
    timeClient.setTimeOffset(currentTimeZone);
    doTimeRepaint = true;  //do a repaint immediately
    Serial.println("*Wortuhr: Saved time zone: " + String(currentTimeZone) + "s (valid until epoch time: " + String(timeZoneValidUntil) + ")");
  } else {
    Serial.println("*Wortuhr: Timezone json invalid! (Status: '" + String(status) + "'");
  }
  wifiClient.stop();
}

[WhymustIhaveaname]

Entschuldigung, I didn't find the reason for your problem, although clues indicate this is the problem of this library. However, I found that you are not the only one, in issue #6 is there a bunch of people suffer from the same problem as you.

May you find the bug.

[WhymustIhaveaname]

You can try my fork. This fork, by default, prints many pieces of information about NTP via Serial, such as the time NTP package is sent and received, the times in the NTP response package. It also changes the return type of update from bool to byte, to record more information(1 on update and success, 0 on update but failed, 2 or 3 on not time to update). These may help you debug.

[FilipDem]

Hi, I have a similar (identical?) problem... I am using always the EPOCH date (and have the offset myself through an online API). However now I see that that the time jumps regularly with one hour... By coincidence the NTP Update interval is one hour... I already did some code reading of the NTPClient, but does not find anything what could explain this... I now added some syslog debug information in case the NTP Update failed (so in case of timeout).

The principle of my implementation is explained below. So I am very interested in the possible results of your debugging. I will also share mine once available.

In the Startup:

    // Start NTP Server connection and wait until time is received
    timeClient.begin();

    // Wait until NTP time received
    uint16_t NTPStart = millis();
    do {
      NTPUpdateDone |= timeClient.update();
      yield();
    } while ( !NTPUpdateDone && (millis()-NTPStart < NTP_TIMEOUT) );
    CurrentDateTime = timeClient.getEpochTime();

And in the Loop:

  boolean NTPUpdate = timeClient.update();
  CurrentDateTime = timeClient.getEpochTime();

Filip

[afrixx]

I did a lot of more testing to find the deviation issue - but I was not successful.

For me I found another solution, which works fine and is much more easy:
The ESP8266 core now contains its own time library: You are able to define multiple NTP Servers, it works in background and there is also a timezone calculation included. So I removed both NTPClient library and Timezone client by adding this functions:
https://github.com/esp8266/Arduino/blob/9913e5210779d2f3c4197760d6813270dbba6232/cores/esp8266/time.c

There is a good example sketch to try out:
https://github.com/esp8266/Arduino/blob/61cd8d83859524db0066a647de3de3f6a0039bb2/libraries/esp8266/examples/NTP-TZ-DST/NTP-TZ-DST.ino

The setting of the timezone is very simple:

  // set timezone to Berlin
  setenv("TZ", "CET-1CEST-2,M3.5.0/02:00:00,M10.5.0/03:00:00", 3);
  tzset();

There you get most important time zone strings:
https://sites.google.com/a/usapiens.com/opnode/time-zones

So... if you're using the ESP, I can only recommend this internal ESP library.

[FilipDem]

Thanks. Was easy to use and seems to work fine. Still to test on long run, but was easy and promising. Don't know why there are not a lot of references to this library when we google ESP8266 NTP... (for people that stick to NTPClient) Yesterday I did bit of code reading again of the NTP client and the only reason I can expect for the time jumps is that the return from the NTP server has zeroes... So time is not updated while the "last_update" time is set as we received a successful answer. Did not debug this, but I am almost sure that this can be the only reason... F

[mitchsf]

My update interval is set to 10 minutes, and the error manifests as 10 minutes behind. The error is highly dependent on the connection. Some ESP32s I have running in other locations, never experience this error. One always shows the error within an hour or two of operation.

I'm going to try a hopefully temporary workaround that will keep trying until the time received is greater than the previous update.

[morres83]

I can also confirm this issue; and the wrong time depends clearly on the update interval or multiples of it.
I think FilipDem is correct with his assumption about "last_update"

[FilipDem]

I stopped using the NTPClient seen I had too much of these problems (don't think there is a lot of activity to solve the concerns).
I switched to the solution of afrixx (see above - default time library). Is easy to integrate and works very well! Can only recommend this (sorry to say this in the NTPClient topic).

[mitchsf]

I seem to have this under control now. There was a bug in NTPClient::update() that was causing it to return true when an update did not occur. That's why it was returning corrupted packets.

The bug has since been fixed: #56

I'm not going to research the cause, maybe it's something I changed inadvertently, or maybe a bug fix. It does make sense that the last good packet received was off by the update interval.

[morres83]

I think the problem lies in the forceUpdate function. The client receives "0", _lastUpdate will be set even though the reception was not correct. Once you read the time by getEpochTime(), you get the wrong offset.

I fixed it in another fork of a similar library, which I use. It combines a check for 0, return codes for update() and a check if the new time is OFFSET later than the current one. You might take a look if you want (ntpUpdate()-function). I think it fixes the issue (as far as I can tell today).

https://github.com/morres83/NTP/blob/master/NTP.cpp

[morres83]

Unfortunately also my Bugfix doesn't solve the problem completely, as I found 1 day later.

Clearly, there is a bug. But I wouldn't call this library user-friendly. But the Arduino core solution and the example sketch clearly is. That's why you cannot find any hint on Google about it.

[mitchsf]

The latest release is working fine for me, at least so far. I added an additional method to get milliseconds. The library referred to by afrixx looks great, but I use the ESP32 and I don't see a version for it.

[afrixx]

The latest release is working fine for me, at least so far. I added an additional method to get milliseconds. The library referred to by afrixx looks great, but I use the ESP32 and I don't see a version for it.

For ESP32 the integrated time library works fine, too:
https://randomnerdtutorials.com/esp32-date-time-ntp-client-server-arduino/