Skip to:

  1. Skip to Jira Navigation
  2. Skip to Side Navigation
  3. Skip to Main Content

Centralized Logging support for DESDL ESPs

Description

With scapps ESPs migrating to HPCC, we need to ensure that the DESDL ESPs are logging transaction info in the Centralized Logging format. This will ensure that the unified ELK stack can work with logs across all components processing a transaction.

  1. Determine how this can be added in a way appropriate for open source

    1. How will it supplant or supplement TxSummary?

    2. Configure the feature on/off?

    3. Flexibility vs standardization of structure and key names

  2. Include support for GlobalID/LocalID/CallerID

Conclusion

None

Activity

Show:

Anthony Fishbeck February 6, 2020 at 2:46 PM

Should make the core capability common code so we can support other native ESP services, not just DESDL.

Anthony Fishbeck February 5, 2020 at 5:43 PM

ah, ok good to hear. Making it readable was helpful.

But also good to get Kevin's feedback.

Kevin Logemann February 5, 2020 at 5:13 PM

We are addressing the issue with filebeats that formerly required the Roxie logged event to be on a separate line.

 

Please note that the 1.2 version of the layout is still being nailed down.   Currently everything is logging the 1.1 version, but the 1.2 version will address some need for the ability to do more complicated reconciliation activities, account for batch/group identifiers, and differentiate types of events being logged.

Terrence Asselin February 5, 2020 at 5:04 PM

I should clarify that I formatted (or attempted to format) these examples for readability. In the actual logs the entries contain no newlines.

Anthony Fishbeck February 5, 2020 at 5:00 PM

JLOG no longer logs newlines without the prefix stamp... the ELK stuff may have relied on this prior but may have to change for roxie anyway.  We should discuss that aspect with Kevin who was looking at the roxie logging.

Fixed
Pinned fields
Click on the next to a field label to start pinning.

Details

Components

Assignee

Reporter

Priority

Compatibility

Major

Fix versions

Pull Request URL

https://github.com/hpcc-systems/HPCC-Platform/pull/14238

Roadmap

Not applicable

Created January 30, 2020 at 7:56 PM
Updated March 30, 2021 at 9:06 AM
Resolved March 30, 2021 at 9:06 AM