Visitor Statistics with Crawler Detection

A plugin for Cotonti CMF that provides detailed tracking of site visits with extended analytics for each visit, and detection of search robots, crawlers, and spiders.
Data is stored in the database and is available for viewing in the administration panel.

big screenshot

Key Features

📊 Collection and Storage of Detailed Information About Each Visit

For every request to the site (excluding administrative AJAX requests — configurable optionally), the following data is recorded in the cot_visitor_stats table:

• Visitor IP address with support for Cloudflare headers (CF-Connecting-IP), X-Forwarded-For, and X-Real-IP.
  If the IP fails validation, 0.0.0.0 is recorded.
• User‑Agent (browser/robot string) — saved in full (up to 500 characters).
• Cotonti user ID (if the visitor is logged in).
• Site page (REQUEST_URI) that the visitor accessed.
• Referer — traffic source (up to 500 characters).
• Date and time of the visit in UNIX timestamp format.

🤖 Detection of Search Bots and Crawlers

A ported version of the Crawler-Detect library (author Mark Beech, MIT) is used.
The plugin does not require Composer — all library classes are located inside the lib/ directory and are adapted to work without namespaces.

Based on signatures (over 100 bots), the visitor is determined to be a robot. When detected, the following is written to the database:

• vs_crawler_name — bot name (e.g., Googlebot, Bingbot, YandexBot, Chrome-Lighthouse, Google-InspectionTool, etc.).
• vs_is_bot — flag 1 (bot) or 0 (human).

The bot list is expandable and can be extended by the administrator by editing the lib/Fixtures/Crawlers.php file.

Additionally, a heuristic is implemented to detect suspicious User‑Agents masquerading as outdated devices (Android 4–6, Nexus 5, Nexus 5X and similar). Such visits are marked as Suspicious UA and are considered bots.

🌍 Geolocation and Network Data

• Country — extracted from the CF-IPCountry header (Cloudflare) or remains empty if the header is missing.
• ISP and VPN/Proxy indicator — obtained through the free ip-api.com API (fields isp, proxy, hosting). The result is cached in the session for 1 hour to avoid exceeding the service limits.
• VPN/Proxy flag (vs_is_vpn) — 1 if ip-api reports that the IP belongs to hosting or a public proxy.

📱 Device and Software Analysis

From the User‑Agent, the following are extracted:

• Browser and version (Firefox, Chrome, Safari, Edge, Opera, IE) — stored in vs_browser.
• Operating system (Windows, macOS, Linux, Android, iOS) — vs_os.
• Device type (Desktop, Mobile, Tablet) — vs_device_type.
• Device model (e.g., Nexus 5, iPhone 13, SM-G973F) — vs_device_model.
  For PCs, PC is indicated.

👤 Unique Visitors

Based on a session variable, it is determined whether the visitor is new or returning.
vs_unique = 1 for the first visit, 0 for subsequent visits within the same session.

🛡️ Bot Whitelist and Blocking of Unwanted Crawlers

The plugin can selectively block bots that are not in the whitelist.
This allows you to:

• Reduce server load by filtering out useless or malicious bots.
• Only allow search robots and services that actually help with promotion (Google, Yandex, Bing, Facebook, PageSpeed Insights, Ahrefs, etc.).
• Protect content from scraping by unwanted crawlers.

How it works:

1. Whitelist (lib/Fixtures/WhitelistBots.php)
   Contains an array of bot names that are allowed access. The check is performed by case‑insensitive substring match.
   The administrator can easily edit this file to add or remove bots.
2. Verification and blocking — performed directly in the global hook (visitor_stats.global.php), which runs on every request.
   If a detected bot is not in the whitelist:
   • The visit is recorded in the database with full detail and the flag vs_blocked = 1.
   • The visitor receives an HTTP response 403 Forbidden (HTML page).
   • Further execution of Cotonti is stopped, saving resources.
3. Function isAllowedBot() (in inc/visitor_stats.functions.php)
   • Always allows humans (null or empty bot name).
   • Blocks “suspicious” UAs marked as Suspicious UA.
   • Compares the bot name against the whitelist using stripos().
4. Debug mode
   In the same file visitor_stats.global.php, there is a $debug flag.
   When set to true, the plugin writes each event to debug_bot.log: time, User‑Agent, bot name, and the decision (ALLOWED/BLOCKED).

🗑️ Automatic Table Cleanup on Schedule

The plugin supports automatic cleanup of all three tables at a specified time interval.
By default, cleanup occurs every 72 hours without the need to configure cron.

How it works:

• Inside the folder plugins/visitor_stats/last_cleanup/, a service file timestamp is created, storing the time of the last cleanup.
• On each request, the plugin checks whether a specified number of hours have passed since the last cleanup.
• If the interval has been exceeded, the tables are cleaned, and the timestamp is updated.
• Auto-cleanup is controlled by the $autoCleanup flag in visitor_stats.global.php (default is true; can be disabled).
• Cleanup does not affect the table structures — only the data is removed.

📈 Administration Panel

In the "Other" section of the Cotonti admin panel, a statistics page is available.
It provides:

• Cards with overall figures: total visits, humans, bots, unique visitors.
• Period filter (days) — currently decorative, will affect data selection in the future.
• "Show only bots" filter — when enabled, the log table displays only rows where a bot/crawler was detected.
• Visit log table with all stored fields, including a "Blocked" column, which shows whether the visit was blocked by the plugin (based on the vs_blocked field).
• Pagination (50 records per page).
• Manual data clear button — completely deletes all records from the three tables without removing their structure. Only available to administrators.

The interface is built on Bootstrap 5.3, which is embedded in the Cotonti admin panel (no external styles need to be connected).

🌐 Localization

Russian and English languages are supported. Localization files are located in lang/.

Plugin Structure and File Descriptions

plugins/visitor_stats/
├── visitor_stats.setup.php # Plugin configuration
├── visitor_stats.admin.php # Administration panel
├── visitor_stats.global.php # Global hook (visit recording, bot blocking, auto-cleanup, debugging)
├── inc/
│ ├── visitor_stats.functions.php # Helper functions and component loading
│ ├── CrawlerDetectService.php # Wrapper service for the CrawlerDetect library
│ ├── VisitorStatsService.php # Main business logic (data collection and recording)
│ └── VisitorStatsRepository.php # Database interaction layer
├── lib/
│ ├── CrawlerDetect.php # Main bot detection class (ported version)
│ └── Fixtures/
│ ├── AbstractProvider.php # Abstract class for signature lists
│ ├── Crawlers.php # Bot and crawler signatures
│ ├── Exclusions.php # Patterns excluded from User‑Agent before checking
│ ├── Headers.php # Headers that may contain the User‑Agent
│ └── WhitelistBots.php # Whitelist of allowed bots
├── setup/
│ ├── visitor_stats.install.sql # SQL queries for table creation
│ └── visitor_stats.uninstall.sql # SQL queries for table removal
├── lang/
│ ├── visitor_stats.en.lang.php # English localization
│ └── visitor_stats.ru.lang.php # Russian localization
├── tpl/
│ └── visitor_stats.admin.tpl # Administration panel template
├── last_cleanup/ # Folder for the automatic cleanup timestamp
│ └── timestamp # File with the time of the last auto-cleanup (created automatically)
└── index.html # Placeholder

Description of Key Components

• visitor_stats.setup.php — plugin metadata: code, name, category, version, author, license, access rights for guests and users.
• visitor_stats.global.php — the central hook that runs on every request. It implements:
  • calling VisitorStatsService::recordVisit() to record a visit,
  • checking bots against the whitelist and blocking unwanted ones,
  • automatic table cleanup based on the timestamp,
  • optional debug logging of blocking events.
• visitor_stats.admin.php — controller for the statistics page in the admin panel. Processes filters, pagination, manual data clearing, builds database queries, and passes variables to the template.
• inc/visitor_stats.functions.php — registers the plugin tables with Cot::$db, loads service classes, and contains a collection of functions for extracting visit information: getRealIp(), getBrowser(), getOS(), getDeviceType(), getDeviceModel(), getCountry(), getIspInfo(), isUniqueVisitor(), as well as the whitelist check function isAllowedBot(). These functions can be used anywhere in Cotonti (including other plugins and templates).
• inc/CrawlerDetectService.php — singleton service providing isCrawler($ua) and getCrawlerName($ua) methods. Each call creates a CrawlerDetect instance with the passed User‑Agent, ensuring up‑to‑date checking.
• inc/VisitorStatsService.php — the main service that aggregates all visit data. The recordVisit() method collects IP, UA, geolocation, bot and uniqueness indicators, and then passes the array to the repository for database insertion.
• inc/VisitorStatsRepository.php — database interaction class. Contains methods for inserting records (insert), getting statistics for a period (countVisits, countBotVisits, countUniqueVisitors), extracting top lists (getTopPages, getTopReferers, getTopCrawlers), and daily breakdowns (getDailyBreakdown). These methods are not yet used in the admin panel but are ready for use.
• lib/CrawlerDetect.php and Fixtures/* — ported bot detection library. Crawlers.php contains the signature list (bot names), Exclusions.php contains patterns removed from the User‑Agent before checking (for speed and reducing false positives), Headers.php lists headers that may carry the User‑Agent.
• lang/ — language files. They contain all text strings used in the admin panel and frontend (if needed).
• tpl/visitor_stats.admin.tpl — statistics page template using Bootstrap 5.3 classes. No external stylesheets are needed since Cotonti already includes Bootstrap.
• setup/*.sql — SQL queries for creating and removing the plugin tables.
• last_cleanup/ — a service folder created by the plugin to store the automatic cleanup timestamp. Contains the file timestamp with the time of the last cleanup.

Database Structure

The plugin works with three tables (the prefix cot_ is defined by the Cotonti configuration; cot_ by default).

Main Table cot_visitor_stats

Stores every recorded session. Field details:

Indexes: PRIMARY (vs_id), KEY on vs_date, vs_ip, vs_user_id, vs_crawler_name.

Daily Summary Table cot_visitor_stats_daily

Unique key on vsd_date.

Crawler Statistics Table cot_visitor_stats_crawlers

Unique key on the combination vsc_date + vsc_crawler_name.

Why Are the cot_visitor_stats_daily and cot_visitor_stats_crawlers Tables Empty?

Currently, the plugin writes only to the main cot_visitor_stats table.
Aggregation of daily totals and crawler summaries into the respective tables is not performed automatically.
These tables were created "for the future": their structure is ready, and future versions are planned to implement a periodic (e.g., via cron) or event‑driven fill mechanism to speed up reports over large time intervals.

The current admin panel builds all statistics directly via SQL queries to cot_visitor_stats (COUNT, grouping), so the summary tables remain empty for now.

Installation

1. Copy the visitor_stats folder into your site's plugins directory.
2. Go to the Cotonti admin panel → "Extensions".
3. Find the Visitor Statistics with Crawler Detection plugin in the list and click "Install".
4. During installation, the SQL queries from setup/visitor_stats.install.sql will execute automatically — three tables will be created.
5. After installation, the plugin starts recording visits immediately (on frontend pages and the admin panel).

To remove the plugin, use the standard procedure in the admin panel: the visitor_stats.uninstall.sql file will be executed, which deletes the tables.

Usage

Viewing Statistics

In the Cotonti admin panel, under the "Other" section, a "Visitor Statistics" item will appear.
The page contains:

• Four cards at the top: total visits, human visits, bot visits, unique visitors.
• Filter form: a period in days can be set (does not affect the selection yet, left for future implementation) and a "Show only bots" checkbox can be ticked — then the table will show only rows where a crawler was detected.
• Log table with paginated output of the latest visits. Columns correspond to all collected fields, including the "Blocked" column, showing the blocking status for each entry.
• Manual clear button (only available to administrators). When clicked with confirmation, it completely deletes all records from the three tables without altering their structure.

Information Functions

Anywhere in Cotonti (templates, other plugins), the following functions are available:

• getRealIp()
• getUserAgent()
• getCountry()
• getBrowser($ua = null)
• getOS($ua = null)
• getDeviceType($ua = null)
• getDeviceModel($ua = null)
• getIspInfo()
• isUniqueVisitor()

They are defined in visitor_stats.functions.php and can be used after the plugin is activated.

Configuring Auto-Cleanup

• Enable/disable: in the file visitor_stats.global.php, find the line
  $autoCleanup = true;
  To disable automatic cleanup, change true to false.
• Change the interval: by default, cleanup occurs every 72 hours.
  To change the interval, edit the condition ($now - $lastCleanup) >= 72 * 3600 in the same file, substituting the desired number of seconds.
• Manual cleanup: at any time, you can use the "Clear all data" button in the admin panel — it resets all tables and updates the automatic cleanup timestamp.

Managing the Bot Whitelist

• Adding a bot: open the file lib/Fixtures/WhitelistBots.php and add a line with a substring of the bot's name to the getAllowed() array.
  For example, for GrokBot: 'GrokBot',.
• Debugging: to check which bots are blocked or allowed, temporarily enable debugging ($debug = true) in visitor_stats.global.php and analyze the debug_bot.log file. Remember to disable debugging after verification.

Technical Details

• Requirements: Cotonti ≥ 1.0 (with PHP 8.x support), PHP 8.0+, MySQL 5.7+ (or MariaDB).
• Crawler‑Detect library ported from the original JayBizzle/Crawler-Detect repository (MIT).
  Changes: namespaces and use statements removed, regular expressions adapted (uses # delimiter instead of /), additional signatures added (Chrome-Lighthouse, Google-InspectionTool, meta-externalagent).
• External dependencies: none. Composer is not required.
• ip-api.com API: used to obtain ISP and proxy indicators. Requests are cached in the session for 1 hour. The service is free but has a limit of 45 requests per minute from a single IP.
• Sessions: the plugin starts a session when necessary to cache ip-api data and to track unique visitors.

Roadmap

• Implement background filling of cot_visitor_stats_daily and cot_visitor_stats_crawlers tables (via cron or delayed script).
• Add charts and visualizations for statistics.
• Introduce a date filter in the admin panel with actual data selection limits.
• Expand the number of detected bots.
• Add data export capabilities (CSV, Excel).
• Integration with Cotonti API to retrieve statistics via REST.

License

The plugin is distributed under the BSD-3-Clause license.
The Crawler-Detect library is MIT.

Author: webitproff.
Repository: https://github.com/webitproff/visitor-stats-crawler-cotonti