Custom Log Formats

OVERVIEW
When Angelfish processes a log file, it needs know the name and position of each field in the log. Angelfish gets this information from the Log Format assigned to each Datasource.

If your log files don't use one of the predefined Log Formats, you'll need to create a custom log format. The Log Format Object can be accessed by Global Admins in Configure - Global - Log Formats.

CREATE A CUSTOM LOG FORMAT
Log formats use a delimiter between fields (e.g. space, comma, colon, semicolon), and individual fields can be surrounded by quotes. Use the dropdown to select a field and click the arrow to move it into the log format box.

Once you create your custom Log Format, copy and paste a single log line into the "Test Log Format" field and click the Test button. If the utility is able to successfully parse the log line and the data & fields match, then the format is correct.

Here's an example of a log line that needs a custom Log Format:

172.18.20.11 urchintools.com - [23/Oct/2021:12:02:03 -0500] "GET /blog/about-urchin-software HTTP/1.1" 200 186 "https://www.urchintools.com/" "Mozilla/5.0+(Windows+NT+10.0;+Win64;+x64)+Chrome/94.0.4606.81+Safari/537.36" "JSESSIONID=a1eea565c4b83f8ba94c8ba2a285c8fb" 23Q12 "f5 failover"

Here's a custom Log Format that will parse the above line. Note how the quote-encapsulated fields in the log line match the quote-encapsulated fields in the custom log format:

%a %h %* %t "%r" %s %b "%R" "%ua" "%c" %* "%*"

If you get stuck, please let us know! Our support team will help you create a custom log format quickly.

LOG FORMAT FIELDS

Property Description
Ignore Field (%*) The contents of the field will be ignored.
Client IP (%a) IP Address of the remote client.
Hostname (%h) Server Hostname.
Username (%u) Username making the requests, present if user is authenticated.
Date (%d) The date of the request.
Time (%T) Time stamp of request.
Custom Date (all) Used in non-standard date stamps - we recommend opening a support ticket with these.
Custom Time (all) Used in non-standard time stamps - we recommend opening a support ticket with these.
Date & Time (%t) A combined date & time stamp, common in Apache logs.
Request (%r) The full request made by the client. Also called cs-request
Request Stem (%U) The Request Stem of the requested resource. Also called Page Stem.
Request Query (%q) The string of data appended to the page stem in the resource request.
Method (%m) The method used to request the resources, i.e. GET, POST, or HEAD.
Status Code (%s) HTTP response status code.
Bytes (%b) The size of the response in bytes.
User Agent (%ua) User Agent of the client
Referrer (%R) The URL of any referring web page linking the client to the request.
Cookies (%c) Contents of the Cookie field
Server IP (%A) IP Address of the machine serving up the resources for the client requests.
Server Name (%v) Name of the server responding to the request.
Port (%p) The port number on which the server serves the request.
Logname (%l) The remote logname (if supplied).
Request Time (%S) The time taken to serve the request, in seconds.
Win32 Status (%Ws) Used in logs from IIS servers and in the "Exclude Win32-Status=64" Preset Filter.
Creation date: 5/26/2022 11:33 AM (Angelfish Support)      Updated: 5/26/2022 12:18 PM (Angelfish Support)