digitalocean/logmatcher

digitalocean/logmatcher

Description: used for matching and filtering syslog/captainslog messages

Language: Go

License: NOASSERTION

Stars: 1

Forks: 0

Open issues: 0

Created: 2023-05-05T04:06:55Z

Pushed: 2023-07-19T18:23:45Z

Default branch: main

Fork: no

Archived: no

README:

Matcher Documentation

The matcher package contains the core expression language for matching captainslog.SyslogMsg data in an intuitive and expressive way.

The [Matcher](matcher.go#L9) interface most importantly provides the method Matches(captainslog.SyslogMsg) bool. There are several implementations of the interface in this package. Those include the following:

• [Hostname](#hostname-matcher)
• [Value](#value-matcher)
• [Facility](#facility-matcher)
• [Severity](#severity-matcher)
• [Timestamp](#timestamp-matcher)
• [KV](#key-value-matcher)
• [UnaryOp](#unary-operator)
• [NAryOp](#n-ary-operator)

From these primitive matchers, essentially any complex expression can be defined. Matchers are used in three forms, and as such, the configuration for each form will be described in their respective sections. Technically there are four commonly used forms, but the fourth (JSON) is simply a special case of another (YAML), so it will not be discussed. The three forms are:

• Golang
• CLI
• YAML

Note: A few of the matchers rely on a special [MatchType](#match-types) struct described below. Please read through that section first.

Match Types

MatchType is essentially an enumerator that represents the many possible comparison operators on which some of the matchers rely. Those types are included here for reference in both their Golang and string-encoded representations:

| Golang | Encoded | |-----------------------|--------------| | _String Types_ | | | ExactMatch | exact_match | | PrefixMatch | prefix_match | | Contains | contains | | Regex | regex | | _Numeric Types_ | | | LessThan | lt | | LessThanEqual | lte | | GreaterThan | gt | | GreaterThanEqual | gte | | _Universal Types_ | | | Equals | equals |

It should become clearer in the following sections how these types are used.

Value Matcher

The Value matcher is intended to match some basic syslog fields such as the program name or content of a message.

Golang

To instantiate a new value matcher in Golang, you can call:

func NewValue(t ValueType, m MatchType, v string) *Value

Here, the ValueType can be one of the following (in Golang and string-encoded forms):

| Golang | Encoded | |---------|---------| | Program | program | | Content | content |

Ex. Usage

v := NewValue(Program, PrefixMatch, "logCatcher_")

CLI

To make this package more accessible to CLI tools, an expression language is provided which exposes the value matchers as functions. e.g:

program(prefix_match, "logCatcher_")

YAML

The YAML encoding of a value matcher is as follows:

---
value_matcher:
type: program
match_type: prefix_match
value: 'logCatcher_'

Hostname Matcher

The Hostname matcher is designed to match the Hostname field found in the header of a syslog message

Golang

To instantiate a new hostname matcher in Golang, you can call:

func NewHostname(m MatchType, n string) *Hostname

Ex. Usage

v := NewHostname(PrefixMatch, "logs-staging-")

CLI

A convenience function is also supplied in the CLI form:

hostname(prefix_match, "logs-staging-")

YAML

The YAML encoding of a hostname matcher is as follows:

---
value_matcher:
type: program
match_type: prefix_match
value: 'logs-staging-'

Facility Matcher

The Facility matcher is essentially a wrapper around the captainslog.Facility class.

Golang

To instantiate in Go, simply call:

func NewFacility(f captainslog.Facility) *Facility

Ex. Usage

f := NewFacility(captainslog.Local6)

CLI

A convenience function is also supplied in the CLI form:

'facility("local6")'

YAML

And in YAML:

---
facility_matcher:
facility: local6

Severity Matcher

The Severity matcher is very similar to the Facility matcher, with the added ability to compare using the equals, less than, or greater than operators.

Golang

To instantiate in Go, simply call:

func NewSeverity(m MatchType, s captainslog.Severity) *Severity

Ex. Usage

s := NewSeverity(LessThan, captainslog.Warn)

CLI

A convenience function is also supplied in the CLI form:

severity(lt, "warn")

YAML

And in YAML:

---
severity_matcher:
match_type: lt
severity: warn

Timestamp Matcher

The Timestamp matcher allows you to match on log times using the equals, less than (before), or greater than (after) operators. Note: LessThanEqual and GreaterThanEqual can also be used, but evaluate the same as their non-equal counterparts.

Golang

To instantiate in Go, simply call:

func NewTimestamp(m MatchType, t captainslog.Time) *Timestamp

Ex. Usage

e := NewTimestamp(Equals, captainslog.Time{
Time: time.Now(),
TimeFormat: time.Stamp,
})

CLI

A convenience function is also supplied in the CLI form:

timestamp(lt, "Jul 13 15:45:30" )

YAML

And in YAML:

---
severity_matcher:
match_type: lt
timestamp: "Jul 13 15:45:30"

Key-Value Matcher

The KV matcher is where things start to get a little more interesting. This structure provides a flexible key-value matcher for arbitrary JSON data in a syslog message.

Golang

To instantiate in Go, call:

func NewKV(key string, m MatchType, value interface{}) *KV

Ex. Usage

kv := NewKV("response.code", LessThan, 300)

This matcher would match a log whose JSON content contains, e.g.:

{"response":{"code":204}}

But not, e.g.:

{"response":{"code":401}}

Periods in the supplied key are used to dereference multi-level JSON data. The supplied value can be anything, but the expectation from the library is that the user supplies proper data types and their corresponding match types.

CLI

A convenience function is also supplied in the CLI form:

kv("response.code", lt, 300)

YAML

Unfortunately, the ability to keep things totally generic breaks down when representing this data in YAML. When supplying numeric rules, you must use the num_value field in the definition, e.g.:

---…