Predefined Log Formats
This article describes the predefined Log Formats in Angelfish. These log formats are available as a formatting option in most web servers.
Predefined Log Formats:
- Auto-Detect
- W3C
- IIS
- Common Log Format (CLF)
- NCSA Combined
- NCSA Combined + Cookie
These appear in the Log Format dropdown menu for each Datasource.
If you're not sure which format your log file uses, please copy a few lines from your log file and paste them into a new support ticket. Our support team can usually figure out the log format after looking at a few lines.
If your log files use a non-standard format, Global Admins can create a custom Log Format in the Log Formats Object.
Help Article: Custom Log Formats
LOG FORMATS
Auto-Detect
When selected, Angelfish analyzes a few lines from the first log file and makes an educated guess about the format. If a match is not found, all logs in the Datasource are skipped and a NOTICE status is shown for the processing job.
We recommend choosing Auto-Detect when creating a new Datasource, unless you're certain the log files use a different Log Format.
W3C
All W3C logs contain a #Fields: header which shows the name and position of each field in the log. The W3C format is commonly used by IIS and by CDN vendors, like Akamai.
Example W3C log file header:
#Software: Microsoft Internet Information Server 10#Version: 1.0#Date: 2021-10-01 00:00:08#Fields: date time s-sitename s-computername s-ip cs-method cs-uri-stem cs-uri-query s-port cs-username c-ip cs-version cs(User-Agent) cs(Referer) cs-host sc-status sc-substatus sc-win32-status sc-bytes cs-bytes time-takenExample W3C Log Line:
2021-10-01 00:00:50 W3SVC2681374 sp-wfe5 10.30.20.11 GET /Pages/FY-Analysis.aspx - 443 corp\csmith 10.50.1.213 HTTP/2 Mozilla/5.0+(Windows+NT+10.0;+Win64;+x64;+rv:91.0)+Gecko/20100101+Firefox/91.0 https://intranet.corp/ intranet.corp 200 0 0 962 2313 21IIS
The IIS log format is a logging option on IIS servers, and uses a fixed format that cannot be customized. Unlike the W3C format, the IIS format does not use the #Fields: directive.
IIS uses W3C by default.
Microsoft's Spec for the IIS format:
COMMON LOG FORMAT (CLF)
CLF logs only contain basic HTTP request data and don't contain the referrer or user agent fields, which limits the amount of data Angelfish can provide. This format is infrequently used, although some products like the Google Search Appliance use a modified version of it.
Field List:
host logname username date:time request statuscode bytesExample Hit:
130.130.255.25 - - [20/Oct/2012:15:03:25 -0700] "GET /index.html HTTP/1.1" 200 1402host: (130.130.255.25)
The remote IP address or hostname of the client that made the request.
logname: (-)
The remote logname (from identd, if supplied)
username: (-)
If logged in or otherwise authenticated this will be the username associated with the client making the request.
date:time: ([20/Oct/2012:15:03:25 -0700])
This field also includes a GMT time offset (-0700)
request: ("GET /index.html HTTP/1.1")
The HTTP request made by the client.
statuscode: (200)
The numeric code indicating the success, failure or redirection of that HTTP request.
bytes: (1402)
The size of the resource being requested by the HTTP request.
NCSA COMBINED
This log format includes the same fields as the Common Log Format, plus the referrer and user agent fields.
Field List:
host logname username date:time request statuscode bytes referrer user_agentExample Hit:
130.130.255.1 - - [20/Oct/2021:15:03:25 -0700] "GET /index.html HTTP/2" 200 1402 "http://www.example.com" "Mozilla/5.0+(Windows+NT+10.0;+Win64;+x64)+Chrome/94.0.4606.81+Safari/537.36"Additional fields:
referrer field: "http://www.example.com"
The URL of the page which linked the client to the site. Angelfish uses this to calculate referral information for a variety of reports.
user_agent field: "Mozilla/5.0+Windows+NT+10.0[...]"
This field contains the browser and platform used by the visitor to the site.
NCSA COMBINED + COOKIE
This log format includes the same fields as NCSA Combined, plus the cookie field.
Field List:
host logname username date:time request statuscode bytes referrer user_agent cookie
Example Hit:
130.130.255.1 - - [20/Oct/2021:15:03:25 -0700] "GET /index.html HTTP/2" 200 1402 "http://www.example.com" "Mozilla/5.0+(Windows+NT+10.0;+Win64;+x64)+Chrome/94.0.4606.81+Safari/537.36" "WRUID=770623994.1162914482"