- Security & Authentication
- System & Performance
- Profiles - Datasources - Filters - Lists
- Users - Collections - Service Accounts
- Data Management & Processing
- Release Notes
Wildcards & Date Substitution
- If your log files don't have a date stamp in the filename, add it!
- Combine Date Substitution & Wildcards in Datasources: u_exYYMM.*
- Date Substitution statements also work in paths: \logs\YYYY\MM\DD\ u_exYYMM.*
- Angelfish uses Regex: .* is a wildcard, not *
Log files usually contain the creation date of the file in the filename. For example, IIS servers create a daily log, like this:
This log was created on May 10, 2022, indicated by "220510" in the filename.
(YYMMDD date format)
(YYMMDD date format)
Moving forward, the filename from May 11 will contain "220511", the filename from May 12 will contain "220512", and so on.
Therein lies the problem. If every new log file created has a different name, how do you configure Angelfish to automatically "see" each new file?
Date Substitution statements automatically adapt to the date stamp . Here's an example:
"YY" represents the current 2 digit year, based on server time
"MM" represents the current 2 digit month, based on server time
"DD" represents the current 2 digit day, based on server time
If today's date is June 1, 2022, the above example matches any file named "u_ex220601.log"
During processing, Angelfish recognizes Date Substitution statements and replaces them with current date information.
However...you usually want to process yesterday's log files, not the current day's log files.
DATE SUBSTITUTION OFFSET
Each Datasource contains a Date Substitution Offset field: this setting adjusts the Date Substitution by a number of days.
By default, the setting is "-1" which subtracts one day from the current date. This extends across months and years.
If the current date is June 1, 2022 and the offset is -1, "u_exYYMMDD.log" matches any file named "ex220531.log" (from May 31).
If the current date is January 1, 2022 and the offset is -1, "u_exYYMMDD.log" matches any file named "ex211231.log" (from December 31).
DATE SUBSTITUTION SYNTAX
YYYY The current 4 digit year
YY The current 2 digit year
MM The current 2 digit month
DD The current 2 digit day
HH The current 2 digit hour
%d The day of the month as a decimal number (range 01 to 31).
%H The hour as a decimal number using a 24-hour clock (range 00 to 23).
%I The hour as a decimal number using a 12-hour clock (range 01 to 12).
%m The month as a decimal number (range 01 to 12).
%y The year as a decimal number without a century (range 00 to 99).
%Y The year as a decimal number including the century.
Angelfish uses the strftime() function, which accepts additional abbreviations. The above are the most commonly-used.
In RegEx, .* roughly translates to "match everything".
But what if you only want to match a certain list of files, like files from 2021? Simply append the Wildcard to the match pattern, like this:
You can also combine Wildcards and Date Substitution statements in the same field, like this:
Date substitution statements are most useful when combined with a wildcard, because they automatically compensate for errors.
For example...if you use u_exYYMMDD.log in a Datasource and your Angelfish server takes a nap over the weekend, you'll have to update each Datasource on Monday so the skipped days are processed.
A better approach is to use u_exYYMM.* - this statement will match any file from the current year and month. If your Angelfish server crashes on the 12th of the month and isn't revived until the 15th, Angelfish will process the logs it missed a few days prior! And yes, you could also use u_exYY.*
You can use other RegEx statement with Date Substitution - see below.
Consider the following list of log files (May 1 - 15, 2022):
If the current date is May 31, 2022:
- matches all files from 2022 (all files listed)
- matches all files from May 2022 (again, all files listed)
- matches a file from May 30, 2022 (assuming an offset of -1, no file exists)
- matches all files from May 10 - May 19, 2022 (the last 6 files in the list)
- matches files from May 1 - May 5, 2022 (the first 5 files in the list)