Help

Overview Admin Chat Data Sync Upload Documents Search Settings Curated Answers Agents, Prompts & Functions Contact Support

Data Sync

Airgentic Help

The Data Sources screen shows your data sources (such as Web Crawl), their sync status, and lets you run a sync manually or on a schedule. This guide describes the Data Sync section and how it fits with crawler settings.

UI Design screen


What is data sync?

Data sync pulls content from your configured data sources and updates the index used by the AI. For a web crawl data source, that means fetching pages, processing them, and indexing the results so the AI can answer questions from your site.


Where to find it

  1. Log in to the Airgentic admin console.
  2. Click the black 'Data Sync' button for the selected service.

You'll see a table of data sources (Web Crawl and JSON Feed), each with Sync Status and Actions, and below that the Web Crawl Schedule and Delete URL sections.


Upload Documents

If you need to index content that isn't available via web crawl—such as internal PDFs, Word documents, or standalone HTML files—use the Upload Documents button at the top of the page. This opens a dedicated screen where you can drag and drop files, manage uploaded documents, and trigger indexing.

See the Upload Documents guide for full details on supported file types, public vs. secure storage, and how indexing works.


Browse Indexed Content

The Browse Indexed Content button at the top of the page shows every unique address (URL) in the search index the AI answers from, across all data sources. Each address is counted once, however many sections it was split into for the AI.

  • The first view lists each website (host) in the index with its number of URLs and a breakdown by content type: web pages, PDFs, Word, Excel, PowerPoint and other files.
  • Click a website to see its top-level folders, then keep clicking to drill down. A folder that is also a page has a link to open it.
  • Addresses that differ only by a query string (such as /events?page=2) or a trailing slash are grouped under the page they belong to and shown as +N variants.
  • Download CSV exports every URL at the level you are viewing, or the whole index from the top level.
  • The data is read from the index when you open the screen and is kept for ten minutes. Use Refresh to read it again after a sync.

Why the count may differ from your sitemap

A sitemap lists the pages its owner chose to publish. The index holds every address the crawler was allowed to reach and could read. The difference usually comes from:

  • Other websites. Subdomains and linked sites inside the crawl scope are indexed too.
  • Query-string variants. Each variant is a separate URL.
  • Documents. PDFs and Office files linked from pages are indexed but rarely listed in a sitemap.
  • Pages missing from the sitemap. Pages that are linked from your site but were left out of the sitemap.

To change what is indexed, adjust the include and exclude patterns in Crawl Scope.


Sync status and actions

Each data source row reports on its own runs only. In the table at the top of the page:

  • Sync Status — Shows when that data source last ran and whether it completed or failed. The Web Crawl row shows how many pages the crawler fetched in that run, and how many are queued, were skipped, or had errors. The fetched count is not the number of pages in the index; use view index or Browse Indexed Content for that. The JSON Feed row shows how many records its feeds returned, how many were new or updated, unchanged or removed, and any feeds that could not be fetched. A data source you haven't set up shows Not configured.
  • Play — Start a sync for that data source on demand. On the Web Crawl row, the dropdown starts a single processing stage (e.g. Process HTML, Index Content). These stages reprocess everything the service has downloaded, so their results appear on the Web Crawl row.
  • Gear — Open that data source's settings. For Web Crawl this is Crawler settings (seed URLs, crawl scope, image extraction, standard result fields); for JSON Feed it is the list of feeds.
  • Logs — View that data source's log. On the Web Crawl row, the dropdown opens the processing stage logs (HTML processing, indexing, etc.).

Only one job runs for a service at a time. While one is running, a bar above the table shows what it is and lets you cancel it.


Web Crawl Schedule

Below the data sources table, Web Crawl Schedule lets you enable or disable automatic syncs and set the time and days of the week. Times use your local timezone. The next run time is shown in the accordion header when the schedule is enabled.


Crawler settings

To control what is crawled (URLs, scope, images, standard result fields), use the gear icon in the Actions column for the Web Crawl row. That opens the Crawler settings screen (General, Crawl Scope, Image Extraction, Standard Result Fields). Configure crawler settings first if you're adding a new site or changing which pages are included.

You have unsaved changes