Collectors

A collector is designed to gather and organize the massive amounts of data required to train or operate AI models (like LLMs), which require high-quality datasets to learn.

Collectors should be as close as possible to the backend storage where the content you want will be collected. For example, if your backend storage is in western europe, you should have a collector also in western Europe. You can use the Collector Status Map to see where you have collectors defined.

To add a colllector:

1. Click Collectors in the navigation pane.
   
2. Click New Collector.
   
3. Enter the following details:
   • Collector Details
     Name – A name to identify the collector.
     URI – The URI to access the collector.
   • Settings
     Default Collector – Slide on to make this collector the default collector.
   • Collector Status
     Collector Status – Select whether you want the status to be Enabled or Disabled.
   • File Processing
     1. Click Connect in the URI field to connect to the collector and retrieve file type mappings.
        
        Only the file types retrieved are indexed for searches.
     2. For each file type, select the Reader if you don't want the default.
        Note

        The list of readers is dependent on the file type.

4. Click Save Changes.

To edit a collector:

1. Click Collectors in the navigation pane.
   
2. Click the collector to edit.
3. Make the required changes.
4. Click Save Changes.