Wildcards & Date Substitution

Creation date: 4/18/2022 10:42 PM Updated: 9/22/2022 2:47 PM

SUMMARY

If your log files don't have a date stamp in the filename, add it!
Combine Date Substitution & Wildcards in Datasources: u_exYYMM.*
Date Substitution statements also work in paths: \logs\YYYY\MM\DD\ u_exYYMM.*
Angelfish uses Regex: .* is a wildcard, not *

INTRODUCTION

Log files usually contain the creation date of the file in the filename. For example, IIS servers create a daily log, like this:

u_ex220510.log

This log was created on May 10, 2022, indicated by "220510" in the filename.
(YYMMDD date format)

Moving forward, the filename from May 11 will contain "220511", the filename from May 12 will contain "220512", and so on.

Therein lies the problem. If every new log file created has a different name, how do you configure Angelfish to automatically "see" each new file?

DATE SUBSTITUTION

Date Substitution statements automatically adapt to the date stamp . Here's an example:

u_exYYMMDD.log

"YY" represents the current 2 digit year, based on server time

"MM" represents the current 2 digit month, based on server time

"DD" represents the current 2 digit day, based on server time

If today's date is June 1, 2022, the above example matches any file named "u_ex220601.log"

During processing, Angelfish recognizes Date Substitution statements and replaces them with current date information.

However...you usually want to process yesterday's log files, not the current day's log files.

DATE SUBSTITUTION OFFSET

Each Datasource contains a Date Substitution Offset field: this setting adjusts the Date Substitution by a number of days.

By default, the setting is "-1" which subtracts one day from the current date. This extends across months and years.

If the current date is June 1, 2022 and the offset is -1, "u_exYYMMDD.log" matches any file named "ex220531.log" (from May 31).

If the current date is January 1, 2022 and the offset is -1, "u_exYYMMDD.log" matches any file named "ex211231.log" (from December 31).

DATE SUBSTITUTION SYNTAX

YYYY The current 4 digit year

YY The current 2 digit year

MM The current 2 digit month

DD The current 2 digit day

HH The current 2 digit hour

%d The day of the month as a decimal number (range 01 to 31).

%H The hour as a decimal number using a 24-hour clock (range 00 to 23).

%I The hour as a decimal number using a 12-hour clock (range 01 to 12).

%m The month as a decimal number (range 01 to 12).

%y The year as a decimal number without a century (range 00 to 99).

%Y The year as a decimal number including the century.

Angelfish uses the strftime() function, which accepts additional abbreviations. The above are the most commonly-used.

WILDCARDS

In RegEx, .* roughly translates to "match everything".

But what if you only want to match a certain list of files, like files from 2021? Simply append the Wildcard to the match pattern, like this:

u_ex21.*

You can also combine Wildcards and Date Substitution statements in the same field, like this:

u_exYYMM.*

BEST PRACTICES

Date substitution statements are most useful when combined with a wildcard, because they automatically compensate for errors.

For example...if you use u_exYYMMDD.log in a Datasource and your Angelfish server takes a nap over the weekend, you'll have to update each Datasource on Monday so the skipped days are processed.

A better approach is to use u_exYYMM.* - this statement will match any file from the current year and month. If your Angelfish server crashes on the 12th of the month and isn't revived until the 15th, Angelfish will process the logs it missed a few days prior! And yes, you could also use u_exYY.*

You can use other RegEx statement with Date Substitution - see below.

EXAMPLES

Consider the following list of log files (May 1 - 15, 2022):

u_ex220501.log
u_ex220502.log
u_ex220503.log
u_ex220504.log
u_ex220505.log
u_ex220506.log
u_ex220507.log
u_ex220508.log
u_ex220509.log
u_ex220510.log
u_ex220511.log
u_ex220512.log
u_ex220513.log
u_ex220514.log
u_ex220515.log

If the current date is May 31, 2022:

u_exYY.*

matches all files from 2022 (all files listed)

u_exYYMM.*

matches all files from May 2022 (again, all files listed)

u_exYYMMDD.*

matches a file from May 30, 2022 (assuming an offset of -1, no file exists)

u_exYYMM1.*

matches all files from May 10 - May 19, 2022 (the last 6 files in the list)

u_exYYMM0[1-5].*

matches files from May 1 - May 5, 2022 (the first 5 files in the list)

Wildcards & Date Substitution

Contents