Search log sample: Google Appliance
Jeremy Bertrand, Interactive Communications Specialist at the Washington State Department of Transportation, has kindly provided a sample of a search log generated by Google Appliance. These queries were searched from this search interface at the Washington State DoT site.
The following legend will help you understand the log data; Ed Costello pointed us to a much better explanation of Google Appliance logs, or you could go directly to Google's documentation.
XXX.XXX.X.103: Searcher's IP address (partially masked to protect privacy)
[10/Jul/2006:14:54:22 -0800]: Query's time/date stamp (-0800 is the zone)
GET /search: request for a search
?access=p: authorization to crawl and index secured content
&entqr=0: unknown
&output=xml_no_dtd: output format
&sort=date%3AD%3AL%3Ad1: sort by date (relevance is the other sorting option)
&ud=1: unknown
&site=AllSites: search scope as configured by site owner to search (in this case, searches all Washington State DoT domains)
&ie=UTF-8: input character encoding (default: utf-8)
&client=www: similar to sourceid (safari, firefox-a)
&oe=UTF-8: output (results) character encoding (default: utf-8)
&proxystylesheet=www: look and feel as defined by site owner
&q=webflow: search query typed by searcher
&ip=XXX.XXX.X.103: searcher's IP address
HTTP/1.1: HTTP protocol (for issuing "GET" command)
200: status of results
3348: # of bytes in results
5: number of results retrieved
0.04: Time spent searching this query
Log sample follows:
XX.XXX. XX.69 - - [10/Jul/2006:10:25:58 -0800] "GET /search?access=p&entqr=0&output=xml_no_dtd&sort=date%3AD%3AL%3Ad1&ud=1&site=AllSites&ie=UTF-8&client=www&oe=UTF-8&proxystylesheet=www&q=shop+drawing+review&ip=XX.XXX.XX.69 HTTP/1.1" 200 8271 77 0.22
XXX.XXX.X.104 - - [10/Jul/2006:10:25:48 -0800] "GET /search?access=p&entqr=0&output=xml_no_dtd&sort=date%3AD%3AL%3Ad1&ie=UTF-8&client=www&q=license+plate&ud=1&site=AllSites&spell=1&oe=UTF-8&proxystylesheet=www&ip=XXX.XXX.X.104 HTTP/1.1" 200 8283 146 0.16
XXX.XXX.X.104 - - [10/Jul/2006:10:25:46 -0800] "GET /search?access=p&entqr=0&output=xml_no_dtd&sort=date%3AD%3AL%3Ad1&ud=1&site=AllSites&ie=UTF-8&client=www&oe=UTF-8&proxystylesheet=www&q=lincense+plate&ip=XXX.XXX.X.104 HTTP/1.1" 200 971 0 0.02
XXX.XXX.XX.130 - - [10/Jul/2006:10:24:38 -0800] "GET /search?access=p&entqr=0&output=xml_no_dtd&sort=date%3AD%3AL%3Ad1&ud=1&site=AllSites&ie=UTF-8&client=www&oe=UTF-8&proxystylesheet=www&q=regional+transportation+governance+commission&ip=XXX.XXX.X.130 HTTP/1.1" 200 9718 62 0.17
XX.XX.XX.14 - - [10/Jul/2006:10:24:13 -0800] "GET /search?access=p&entqr=0&output=xml_no_dtd&sort=date%3AD%3AL%3Ad1&ie=UTF-8&btnG=Search&client=www&q=noise+barrier&ud=1&site=AllSites&oe=UTF-8&proxystylesheet=www&ip=XX.XX.XX.14 HTTP/1.1" 200 9429 416 0.18
XXX.XX.XX.76 - - [10/Jul/2006:10:24:13 -0800] "GET /search?entqr=0&access=p&output=xml_no_dtd&sort=date%3AD%3AL%3Ad1&ie=UTF-8&btnG=Search&btnG.y=9&client=www&q=lrfd+design+examples&btnG.x=18&ud=1&site=AllSites&oe=UTF-8&proxystylesheet=www&ip=XXX.XX.XX.76 HTTP/1.1" 200 8951 34 0.35
XX.XX.XX.186 - - [10/Jul/2006:10:23:47 -0800] "GET /search?access=p&entqr=0&output=xml_no_dtd&sort=date%3AD%3AL%3Ad1&ie=UTF-8&btnG=search&btnG.y=0&client=www&q=record+of+decision&btnG.x=0&ud=1&site=AllSites&oe=UTF-8&proxystylesheet=www&ip=XX.XXX.XX.186 HTTP/1.1" 200 9175 454 0.14
XX.XXX.XXX.194 - - [10/Jul/2006:10:23:08 -0800] "GET /search?entqr=0&access=p&start=10&output=xml_no_dtd&sort=date%3AD%3AL%3Ad1&ie=UTF-8&btnG=Search&client=www&q=memorial+interchanges&ud=1&site=AllSites&oe=UTF-8&proxystylesheet=www&ip=XX.XXX.XXX.194 HTTP/1.1" 200 9286 78 0.25
XX.XX.XX.194 - - [10/Jul/2006:10:22:29 -0800] "GET /search?access=p&entqr=0&output=xml_no_dtd&sort=date%3AD%3AL%3Ad1&ie=UTF-8&btnG=Search&client=www&q=memorial+interchanges&ud=1&site=AllSites&oe=UTF-8&proxystylesheet=www&ip=XX.XXX.XXX.194 HTTP/1.1" 200 8551 78 0.10
Comments
HTTP/1.1 is the protocol (i.e., it otherwise could be HTTP/1.0). So, this log entry should indicate that the client used to HTTP/1.1 to issue the GET /search request.
Posted by: Jay Fienberg | July 20, 2006 12:06 PM
Thanks Jay!
Posted by: Lou Rosenfeld | July 20, 2006 6:45 PM
Thank you for this excellent example. I am looking forward to the book, and to reading the draft chapters too.
Question: Would a search analytics tool be able to clearly show which, if any, search result was clicked on, or if the visitor explored more than one? Would that data be part of an IP-based behavioral tracking system?
Posted by: Melody | July 24, 2006 10:33 AM
Hi Melody; I think it partly depends on what the search engine (or analytics tool) captures--or doesn't--in a search log. The reporting tool really depends on this data to be captured before it reports on it. We're hoping to post examples of additional search logs here; hope to show you more soon!
Posted by: Lou Rosenfeld | July 24, 2006 10:42 AM
?access=p: unknown
There are authorization integration features in the appliance that allows you to crawl and index secured content. It looks as though that this info is passed to the results as well to tell you that it could be either in the public or perhaps secured.
&ie=UTF-8: unknown; browser encoding?
Required for all internet applications to identify character encoding of app. One could be the input encoding and the other is the output encoding.
Another person to touch base would be Avi Rappaport.
Posted by: Madonnalisa Chan | July 24, 2006 12:34 PM
Our company has customized Urchin software to process logs created by the Google Search Appliance and the Google Mini. We feel that many of the common analytics applications are adequate for understanding usage when these devices are used on an external facing website. However, search analytics for an intranet are a bit different. That's why we've created the following reports:
* Traffic Summary
* Sessions Graph
* Searches Graph
* IP Addresses
* IP Drilldown
* Depth of Sessions
* Length of Sessions
* Search Terms
* Keyword Progressions
* Keyword To and From
* Exit Keywords
* Search Term Number of Results
* Zero Result Searches
* Bytes Graph
* Status and Errors
* Collections by Hits
* Front End by Hits
* Processing Time by Hits
Posted by: Justin Cutroni | July 25, 2006 12:16 PM
Justin, your reports sound very useful. Any chance we could see an example report or two? I realize that if it's for intranet purposes this might be sensitive, but if you could share even an example of keyword progressions or one of the graphs that'd be really cool.
Posted by: Rich Wiggins | July 27, 2006 3:40 PM
Thanks Madonnalisa and Justin!
Posted by: Lou | July 31, 2006 11:44 AM