Is using %uuid{RANDOM} pattern in JsonTemplateLayout considered garbage-free?

[ViliusS]

So if I use such JSON template layout, is it garbage-free?

{
  "logging.googleapis.com/insertId": {
    "$resolver": "pattern",
    "pattern": "%uuid{RANDOM}",
    "stackTraceEnabled": false
  }
}

What about %uuid{TIME} ?

[ppkarwasz]

Neither one is GC-free. Each one will create an instance of UUID and String. Scalar replacement might occur, when the JVM is hot enough, but you should test it on a concrete version of the JVM.

[ppkarwasz]

BTW: Using %uuid as value for logging.googleapis.com/insertId does not fulfill the unicity requirement from the documentation:

Optional. A unique identifier for the log entry. If you provide a value, then Logging considers other log entries in the same project, with the same timestamp, and with the same insertId to be duplicates which are removed in a single query result. However, there are no guarantees of de-duplication in the export of logs.
If the insertId is omitted when writing a log entry, the Logging API assigns its own unique identifier in this field.

The same log event, delivered by different appenders will have different UUIDs. See #3503 for a discussion.

[ViliusS]

BTW: Using %uuid as value for logging.googleapis.com/insertId does not fulfill the unicity requirement from the documentation:

Optional. A unique identifier for the log entry. If you provide a value, then Logging considers other log entries in the same project, with the same timestamp, and with the same insertId to be duplicates which are removed in a single query result. However, there are no guarantees of de-duplication in the export of logs.
If the insertId is omitted when writing a log entry, the Logging API assigns its own unique identifier in this field.

The same log event, delivered by different appenders will have different UUIDs. See #3503 for a discussion.

I'm going to propose PR for GcpLayout.json template shortly. Among other things, counter cannot be used with logging.googleapis.com/insertId either because it produces duplicate IDs for different JVM threads. It also restarts from ID 1 if Java application is restarted.

I think having different UUID for different appenders are exactly how it is supposed to work. Different appenders pushing to the same Google Cloud Log would probably mean different log event formatting anyway, so we should ensure that such messages are not de-duplicated by GCP.

Alternatively, insertId could be omitted completely, so GCP would generate their own IDs.

[ppkarwasz]

Alternatively, insertId could be omitted completely, so GCP would generate their own IDs.

This is probably the route I would choose. The only purpose of this value is to deduplicate log events with the same timestamp. Short of creating a ContextDataProvider with a counter, I don't see a reliable way to do that.

Log duplication shouldn't occur anyway, except misconfigurations like:

<Loggers>
  <Root level="INFO">
    <AppenderRef ref="GCP"/>
  </Root>
  <!-- Additivity is intentionally not disabled to create duplicates -->
  <Logger name="org.apache.logging">
    <AppenderRef ref="GCP"/>
  </Logger>
</Loggers>

[rgoers]

I hate to disagree Piotr, but our time based uuid exists for exactly this use case. It will provide a unique value across all servers. While it isn’t going to be unique forever, the length of time it is guaranteed to be unique should be sufficient for virtually any use case.

[ppkarwasz]

I am not saying that %uuid will give the same value for different log events, but that it will give different value for the same log event if they are delivered by different appenders.

GCP says:

If you provide a value, then Logging considers other log entries in the same project, with the same timestamp, and with the same insertId to be duplicates which are removed in a single query result.

Since each time the same log event is formatted, another UUID is generated, every insertId will be different and no duplicate events will be merged.

[rgoers]

I would need to read more on the subject but I have to assume this is very similar to the problem I faced with Flume. It guarantees delivery but it doesn't guarantee an event won't be delivered multiple times. This is solved by providing a unique id. Log4j's TimeBased UUID solves this problem by adding a unique id when the event is created. Whether there are multiple appenders or not only matters depending on where they are trying to deliver the data. However, it definitely makes more sense to ensure the UUID is injected into the LogEvent before any Appenders process it.
I am not sure why you are assuming only multiple Appenders can cause duplicates to occur.

[ppkarwasz]

However, it definitely makes more sense to ensure the UUID is injected into the LogEvent before any Appenders process it.

This is the main problem: %uuid is a converter and it can not modify the LogEvent. If the same log event is formatted multiple times, each time

I am not sure why you are assuming only multiple Appenders can cause duplicates to occur.

The question is probably, where the duplication occurs. If a log event is passed to multiple appenders (or twice to the same appender), then %uuid will not be able to identify the duplicates. This was discussed in #3503.

If the duplication occurs in the appender logic (e.g. it is cause by network retries), then sure, %uuid will help.