Custom Log Formats
OVERVIEW
When Angelfish processes a log file, it needs know the name and position of each field in the log. Angelfish gets this information from the Log Format assigned to each Datasource.
If your log files don't use one of the predefined Log Formats, you'll need to create a custom log format. The Log Format Object can be accessed by Global Admins in Configure - Global - Log Formats.
To see what a Log Format is supposed to look like, open an existing Log Format:
"NCSA Combined + Cookie" is a good example.
CREATE A CUSTOM LOG FORMAT
To create a new Log Format, select a field from the Log Fields dropdown menu and click the arrow to add it to the Log Format field.
Each field can be separated by a space, comma, or a tab (use \t to represent tab). This is called the field delimiter.
Individual fields can be encapsulated with double quotes, brackets, or other characters. You'll need to manually enter these in the Log Format field.
Once you create your custom Log Format, copy and paste a single log line into the "Test Log Format" field and click the Test button. If the utility is able to successfully parse the log line and the data & fields match, then the format is correct.
Here's an example of a log line that needs a custom Log Format:
172.18.20.11 urchintools.com - [23/Oct/2021:12:02:03 -0500] "GET /blog/about-urchin-software HTTP/1.1" 200 186 "https://www.urchintools.com/" "Mozilla/5.0+(Windows+NT+10.0;+Win64;+x64)+Chrome/94.0.4606.81+Safari/537.36" "JSESSIONID=a1eea565c4b83f8ba94c8ba2a285c8fb" 23Q12 "f5 failover"Here's a custom Log Format that successfully parses the above line. Note how the quote-encapsulated fields in the log line match the quote-encapsulated fields in the custom log format:
%a %h %* %t "%r" %s %b "%R" "%ua" "%c" %* "%*"If you get stuck, please let us know! Our support team will help you create a custom log format quickly.
LOG FORMAT FIELDS
| Property | Description |
|---|---|
| Ignore Field (%*) | The contents of the field will be ignored. |
| Client IP (%a) | IP Address of the remote client. |
| Hostname (%h) | Server Hostname. |
| Username (%u) | Username making the requests, present if user is authenticated. |
| Date (%d) | The date of the request. |
| Time (%T) | Time stamp of request. |
| Custom Date (all) | Used in non-standard date stamps - we recommend opening a support ticket with these. |
| Custom Time (all) | Used in non-standard time stamps - we recommend opening a support ticket with these. |
| Date & Time (%t) | A combined date & time stamp, common in Apache logs. |
| Request (%r) | The full request made by the client. Also called cs-request |
| Request Stem (%U) | The Request Stem of the requested resource. Also called Page Stem. |
| Request Query (%q) | The string of data appended to the page stem in the resource request. |
| Method (%m) | The method used to request the resources, i.e. GET, POST, or HEAD. |
| Status Code (%s) | HTTP response status code. |
| Bytes (%b) | The size of the response in bytes. |
| User Agent (%ua) | User Agent of the client |
| Referrer (%R) | The URL of any referring web page linking the client to the request. |
| Cookies (%c) | Contents of the Cookie field |
| Server IP (%A) | IP Address of the machine serving up the resources for the client requests. |
| Server Name (%v) | Name of the server responding to the request. |
| Port (%p) | The port number on which the server serves the request. |
| Logname (%l) | The remote logname (if supplied). |
| Request Time (%S) | The time taken to serve the request, in seconds. |
| Win32 Status (%Ws) | Used in logs from IIS servers and in the "Exclude Win32-Status=64" Preset Filter. |