English
Magyar
Server logs often reveal requests that aren’t shown in traffic reports. Analysing web server logs reveals which pages or files were requested, when the request was made, and what HTTP response code was returned. A 200 code indicates a successful request, whilst a 404 code indicates that the requested page does not exist.
If you’re looking for an error or want to understand what search engine bots are crawling, the web server log provides detailed information. Each entry records a request received by the server, including requests from bots. The user-agent field can help you identify the type of request, but on its own it does not prove that the request was actually sent by the bot in question.
Below, you’ll learn about the key fields in log entries and the meanings of the most common HTTP codes, as well as how server logs differ from web analytics. We’ll look at how they can be used when analysing bot traffic, and when it’s best to use a spreadsheet program or a log processing tool. This will give you a clearer picture of what is happening on the server and which errors require attention.
Checking HTTP response codes
The web server sends an HTTP response to the browser, and the status code indicates what happened with the request. Interpret the code in the log alongside the relevant URL, timestamp and request.
From a GDPR perspective, an IP address or other identifier in a log may contain personal data. Therefore, the purpose of data processing, as well as access and retention, must be managed in a restricted and transparent manner. In Hungary, too, these data protection considerations govern the handling of web server logs.
The risks associated with log storage include unauthorised access and retention for too long. Minimise these risks by restricting access rights, protecting stored data and removing unnecessary log data.
HTTP responses and status codes
An HTTP response is the server’s reply to a request from a browser or other client, and the status code summarises the result of this response. A 200 code indicates a successful request, whilst a 301 indicates a permanent redirect and a 302 indicates a temporary redirect. Checking these helps to determine whether the user is reaching the expected page or is following an incorrect redirect path.
Analysis of HTTP error codes
Analysing HTTP error codes helps to identify the types and frequency of faulty requests in log entries. According to the Internet Assigned Numbers Authority (IANA), 4xx status codes indicate a client error, whilst 5xx status codes indicate a server error, so these can be used to measure the distribution of errors. A 404 code indicates a requested page does not exist, whilst a 503 code indicates a server that is temporarily unavailable. You should also examine the relevant URLs and timestamps.
In practice, first group the entries by status code, then identify which files or pages are duplicated. For example, a 404 response may result from a typo, removed content or a broken internal link. Recurring 503 responses, on the other hand, may indicate a server-side availability issue. The status code alone does not reveal the underlying cause, so examine it alongside the timestamp and the relevant request.
The ideal HTTP response code for website pages is 200 or 301. If the old address of a moved page returns a 404 error, check the redirect rule. Meta refresh redirects are not recommended from an SEO perspective; instead, a permanent 301 redirect is advised. A poorly targeted redirect can still mislead the user even if no error code is generated, so please also check the destination page.
Apache HTTP Server logs
The Apache HTTP Server access log can be used to analyse requests and responses. On Debian and Ubuntu systems, the default log location is /var/log/apache2/access.log, whilst on RHEL, Red Hat, CentOS and Fedora systems it is /var/log/httpd/access_log. On FreeBSD systems, the file is located at /var/log/httpd-access.log. Knowing the correct path helps you quickly find the entry for a specific request.
Look up the relevant time, URL and HTTP status code in the Apache log, then compare them with similar requests. For example, if the same page repeatedly returns a 404 error, check whether the page’s URL has changed or whether the file is actually accessible. Among other things, the temporal pattern of recurring 5xx responses can also help distinguish between an occasional error and a persistent server issue.
Identifying search engine bots and the limitations of user-agents
The web server receives and processes incoming HTTP requests, then sends a response to the client. From the log entries, you can see which pages were requested by search engine bots, how often they returned, and how long the crawling took.
Search engine crawlers indexing websites
Analysing how search engine bots crawl a website involves reviewing the requests they send to the server. It shows which URLs are being crawled and helps you spot if important content is being missed or if attention is being directed to incorrect URLs. The log can record every request that actually reaches the server, including those from bots.
Search for entries based on the user-agent field, then compare the timestamps and the requested URLs. For example, if a bot regularly requests the same address, check whether the page is accessible and whether the server returns the expected response. If the log also contains requests from users’ browsers, the nature and frequency of these requests can help you distinguish between different types of traffic.
Analysis of search engine crawler activity
The crawl analysis shows the crawl pattern based on the pages requested by the bots, the frequency of requests and the processing time. This is useful because it reveals what the bot is focusing on and where technical issues may arise. The log may also show the duration of the crawl in milliseconds.
Group requests by URL families, such as product pages, images or search results. Then check whether the bots return to the same pages and reach the important content. If you see a lot of invalid or low-value URLs in the log, you can use the entries to find out which URLs the requests are directed at. The pattern also shows whether the site crawl consists of quick, short requests or longer server processes.
The limitations of user-agents
The user-agent is a text identifier included in an HTTP request that identifies the browser or software acting as a robot. It is a useful filtering criterion, but on its own it does not verify the genuine origin of the request. The text ‘Googlebot’ appearing in the log entry is therefore merely an indication, not conclusive proof that the request was actually sent by a Google bot.
When evaluating, do not just take this field into account. Compare the paths of requests marked with the identifier, their distribution over time and the server’s responses. If the user-agent indicates a bot, but the requests repeatedly access unusual pages, treat this as traffic requiring further investigation. You can certainly determine from the log what HTTP request the web server received and what response it sent, but you cannot necessarily determine who was behind the request.
Together with log analysis software, you can filter user-agent values, URLs and timestamps. This allows you to compare requests identified as coming from bots with the pages actually requested and the response times. Analysing web server logs makes it possible to map out browsing patterns, whilst identifiers must still be interpreted with caution.
Log processing tools: spreadsheet programs, Splunk, Sumo Logic and the Elastic Stack
To analyse Apache and Nginx logs, first locate the relevant file, then select a tool appropriate for the volume of the log. Smaller files can be filtered using a spreadsheet program; for processing large volumes of web server access data, a log analysis system is recommended.
When analysing Apache and Nginx logs, first identify the file, then select a time period, filter by URLs or status codes, and finally compare any recurring entries. For Apache, the location of the log depends on the system; for Nginx, the default directory is var/log/nginx/.
You can also process and segment smaller log files in Excel. Splunk, Sumo Logic and the Elastic Stack are log processing tools that can help you make sense of larger volumes of data. The choice depends on the size of the log file, as well as the required filtering, searching and aggregation.
A spreadsheet for small datasets. If the format allows, organise the log fields into columns in Excel. Then filter the rows by, for example, URL or HTTP status code, and tally up the duplicates. This allows you to quickly see which entries relate to a selected file or page. With large datasets, manual sorting and filtering becomes cumbersome, so it is advisable to use a specialised log analyser.
Specialised systems for larger datasets. Splunk, Sumo Logic and the Elastic Stack can be used to search, filter and visualise logs in a clear and organised manner. For example, if you are searching for a specific URL, you can compare the requests recorded in the Apache or Nginx logs. Filtering by status code helps to distinguish between failed and successful responses. When selecting a tool, consider how the data is ingested, which fields you can search, and what kind of aggregations you require.
The method of processing also depends on the file format. In a spreadsheet program, incorrect interpretation of field delimiters or quotation marks can cause data to be misaligned. Check against a few original log entries to ensure that the timestamp, the requested URL and the status code have been placed in separate fields. In search systems, consistent field names make Apache and Nginx entries comparable.
Start your analysis with a specific question: which requests were sent to a particular page, and how did the web server respond to them? Then expand your search to other URLs or time periods. This will allow you to compare the error detected in the browser with the corresponding log entry, making it easier to link your observations to a specific server response.
Adjust the processing method to suit the size of the log file. You can read and segment smaller log files using standard data processing programs, such as Excel. First, check that the fields are correctly separated, then sort and filter the data according to the criteria relevant to the issue under investigation. The overview remains more manageable if you do not immediately extend the search to every entry, but instead focus on a selected URL, response code or time period. Compare the results with the original log entries as well; this will make it easier to spot any misinterpretations of the fields. When dealing with large volumes of server access data, a specialised tool is required rather than manual analysis. The search and filtering options offered by these tools can help you review the data, whilst standardised field names make it easier to compare Apache and Nginx entries. Before embarking on a more extensive analysis, define the question you are seeking to answer and check whether the selected fields provide sufficient information. If not, adjust the search or aggregation, then re-examine the relevant entries. In this way, you can tailor the method to the size of the log and the purpose of the analysis.
Interpreting the fields of the Common Log Format
Analysing web server logs involves examining request log entries, which show what data the server has served. Log files may record the date, time, time zone offset, IP or DNS address, request, response code and data size. You can estimate traffic by aggregating the requests; you can only measure response time if the log contains this field. The error rate can be calculated by comparing the number of failed requests with the total number of requests.
The fields of the Common Log Format
The Common Log Format is a logging standard used by web servers, which displays requests in a standardised field order. This allows you to look up who requested what content and when, and then compare the request with the server’s response.
In an entry, the address may be an IP or DNS address, whilst the time may be displayed with a time zone offset. The request field may contain the path and the protocol used. In addition to the response code, the size of the served object may also be listed in bytes. By examining these fields together, you can distinguish, for example, between a page request and the download of a larger file.
When reading the format, proceed from left to right: first identify the timestamp and the requested URL, then the response code. Next, check the data size. If you see an unexpectedly large value in a line, check which page or file it relates to. The exact set of fields may also depend on the logging settings.
The data content of web server logs
The web server log consists of unique request lines with timestamps, thus showing server-side access to pages and files. This makes troubleshooting easier, as the entries can be analysed without JavaScript code or cookies.
The log may contain the visitor’s IP or DNS address, the time of the request, the requested resource, the transfer protocol, the amount of data transferred and the browser type. Web server logging may also record files stored on the server and subsequently downloaded. This makes it possible, for example, to check whether the server served an image or a document.
A log entry does not necessarily correspond to a single person or visit. The same IP address may be used by several visitors, and dynamic IP addresses may change over time. Furthermore, access via a browser’s cache or a proxy server may not necessarily appear in the log that records direct server accesses. Therefore, the number of log entries alone does not reflect the number of unique visitors.
Traffic, response time and error rate
The number and distribution of requests can be summarised from the log, providing a server-side view of visitor traffic. However, the total number of requests is not the same as the number of unique visitors, as loading a single page can trigger multiple file requests.
To analyse response times, you need a log field that records the duration of the request. The Common Log Format does not include such data in its basic fields, so please check whether the web server’s extended logging records this information. If the duration is included in the log, you can use URL-based grouping to identify which pages take longer to serve.
To calculate the error rate, compare the number of error responses with the total number of requests during the same period. 4xx codes indicate a client error, whilst 5xx codes indicate a server error. For example, counting 404 responses separately from the total number of requests shows the proportion of ‘page not found’ errors. Break the result down further by URL and date to see whether the error is linked to a single address or multiple pages.
The differences between server logs and web analytics data
The server log records individual requests received by the web server, whilst web analytics aggregates visitor behaviour. Logs provide direct data on the content served by the server, which is why requests from bots and file downloads may also appear in them. Analytical measurement, by contrast, often uses JavaScript code or cookies.
Retain server logs for as long as is necessary for troubleshooting and operations. Storage requirements increase in line with traffic and the volume of logged requests. On high-traffic websites, files can accumulate rapidly, so additional storage resources may be required.
The two data sources answer different questions. The log shows what requests the web server received and what responses it provided. Web analytics can also display behavioural metrics such as the bounce rate, which the server log does not provide on its own. Logs therefore complement the analysis, but they are not the sole source for examining visitor behaviour.
The log file consists of individual request lines, each with a timestamp. Opening a single page can result in multiple lines, as the browser may send separate requests for images, style sheets and other files, for example. Therefore, the number of lines or requests is not, in itself, the same as the number of page views or unique visitors. Requests from bots may also appear in the web server log.
The log can also reveal server-side events that are not recorded by JavaScript-based monitoring. Analysing served files and direct requests, for example, can help determine whether the server has served a PDF document or an image. In contrast, requests originating from the browser’s cache or via a proxy server do not necessarily appear in the log that records direct server accesses.
Dynamic operations within the site do not always appear as separate log entries. An AJAX operation may be recorded in the server log if it triggers an actual server request; changes carried out solely within the browser do not constitute a new server request. To gain a more complete understanding of visitor activity, it is therefore advisable to interpret the log and web analytics data together.
Even IP addresses cannot always be used to determine the exact number of visitors. Dynamic IP addresses may change, whilst the same IP address may be used by several users. Visitor statistics derived from log traffic should therefore be interpreted with caution. To interpret the data reliably, treat requests, visits and estimates of unique users separately.
When conducting a practical analysis, first determine whether you wish to analyse technical errors, file serving or visitor behaviour. Then select the appropriate data source: use logs to analyse error codes and server requests, and web analytics for behavioural metrics. By aligning the time period and the pages concerned, you can compare the two views without treating the number of requests as equivalent to the number of visitors.
When analysing web server logs, it is important to bear in mind that the number of requests is not the same as the number of unique visitors. Intra-page AJAX operations do not necessarily appear in the log file if they do not involve an actual server request; according to aWh’s experts, traffic data in log files may be inaccurate in the case of dynamic IP addresses or multiple visitors using the same IP address.
aWh: Web hosting
aWh offers web hosting for running websites. If, based on log analysis, you need to review your hosting environment or available resources, choosing the right hosting plan may be the next step.
The right solution depends on your needs. Web hosting may be suitable for running a website, whilst a KVM VPS provides a different server environment. aWh’s range of services also includes domain registration and Windows VPS. When making your choice, consider the environment in which your website runs and the tasks you wish to carry out.
Take a look at aWh’s web hosting solutions, or visit www.awh.hu.
Turn log analysis into a standard practice
Analysing web server logs helps with decision-making when specific actions can be derived from your observations. Make a note of which errors recur, which requests are unusual, and how the log pattern changes following each modification. This allows you to compare the server’s performance and search for the causes of problems in a more targeted manner.
Establish a process whereby only authorised personnel have access to the logs, and ensure that analysis is tailored to operational needs. If, based on the results, you decide to review your website’s technical environment, consider what resources and configurations are required to ensure stable operation.
If you’d like to review the next technical steps, please get in touch with aWh. A systematic troubleshooting process, starting with the log entries, can help you plan more effectively.
Frequently asked questions about server logs
How do server logs work?
The web server logs events relating to incoming requests, and the administrator can use these for troubleshooting or analysing operational patterns. If you have made a change following an error, make a note of the time, then compare the entries generated before and after the change. A consistent time zone and the logging of changes make it easier to link events together.
Where are the logs?
You can find the logs in the hosting control panel, the server’s file system or the logging interface of the operating environment. Look for access and error logs, then check whether there are any archived or compressed files alongside the current file. Older logs are often stored in a separate file. If you are using a control panel, its log viewer may display the available entries.
What is a log file?
A log file is a data file that records events for later analysis. In a web server environment, separate access and error logs may be generated: the former helps you track requests, whilst the latter helps you review problems reported during operation. When troubleshooting, first decide which event you are looking for, then select the appropriate log. A copy of the original file preserves the initial state for the investigation.
What do bots want?
The purpose of bots depends on how they operate: some collect content for search engine indexing, whilst others automatically monitor or retrieve data. It is not always possible to determine the creator’s intent from the log file alone, so it is better to examine the impact on the website. Check whether the requests are overloading resources or accessing paths that should not be public.
What tools and software can be used to analyse web server logs?
Adapt the tool to the log’s format and processing method. In a spreadsheet, first check that the fields are properly separated, then use filters and sorting. With Splunk, Sumo Logic or Elastic Stack, standardising the fields makes searching easier. When working in a team, access control, the sharing of queries and the retrievability of archived data are also important considerations.
How can you identify 4xx and 5xx HTTP errors, and what are their most common causes?
Search the log for requests associated with incorrect responses and group them by path, time or affected function. A recurring 404 error, for example, may indicate an outdated link or removed content; check whether the URL has changed. If several pages are returning errors at the same time, compare the occurrence with any configuration or content updates to narrow down the possible causes.
How can you identify bots, attacks and suspicious traffic from the logs?
It analyses suspicious patterns by cross-referencing multiple log sources, such as web requests against authentication events. Particular attention should be paid if there are a large number of attempts targeting protected functions within a short period, or if a series of errors appears following an unusual request. Retain the original logs for the period under investigation, restrict access, and document your observations with timestamps.
What free and paid log analysis services are available, and how much do they cost?
The tools include spreadsheet programs and specialised log processing systems, such as Splunk, Sumo Logic and Elastic Stack. Don’t choose a tool based solely on its name: consider how the logs are ingested, what formats need to be processed, and who will have access to them. It is also worth defining your search, archiving and alerting requirements in advance to ensure the tool fits in with day-to-day operations.