Need Support? We're Here to Help with Web Scraping Solutions

About Crawler Manager

Crawler Manager is an intuitive web-based platform developed by CrawlerHub that allows you to take control of your web scraping projects. It's built to give you real-time visibility, access to your data, and seamless collaboration with our developers.

To initiate your journey with CrawlerHub, please reach out to us through our Contact page and share your data needs. Our sales team will review your requirements and send a tailored proposal based on your use case, data volume, website complexities, and/or extraction frequency.

• Set up a project and configure your crawlers
• Run a sample to ensure data quality and accuracy
• Upon approval of the sample, we will then begin full-scale extraction

How to Get Started

To access Crawler Manager, all you need to do is onboard with us — either by subscribing to our services or confirming a data project.

Sign In

To access your Crawler Manager account:

• Visit: www.app.crawlerhub.com Or go to crawlerhub.com > Crawler Manager > Click Sign In
• Click on the "Sign In" button
• Enter your email address and password
• Click on the "Sign In" button

Create a New Project

To create a new project in Crawler Manager:

• On the Dashboard, click on New Project
• Or go to Projects page by clicking on Manage Project from Dashboard, and click on + New Project located on top left.
• A new window will open, asking for necessary details.
• Enter details:
• Project Name
• Description: Provide your data expectations, volume, format, and other necessary details. Data Source such as web URL
• Start Date, Frequency and appropriate expected Deadline.
• Data Points to specify
• Team Members if you want
• Click on Create Project to save.
• Your new project appears instantly in the list and will undergo review by developers for crawler development.

Manage a Project

To dive into managing an existing project:

• Click any project in the Manage Project list.
• You'll land on the Project Details page, where you can:
  • View key details at the top, like description, owner, deadlines, frequency, and current status.
  • See recent crawl activity with run dates, times, and crawler status.
  • Check project statistics (total records, download size).
  • Manage your team members.

Edit Project's Detail

To edit a project's information:

• Inside the selected project, click the Edit Project button.
• Make your changes, for example:
  • Update the project name or description
  • Change the owner or update deadlines and frequency
• Click Update Project to save changes.

Add or Remove Team Members

To add a team member:

• Inside the project details page, scroll to the Team section.
• Click the Add button.
• Enter the user's email address and hit Enter.
  • If the user already exists and is verified:
    • They'll be added to the project immediately.
  • If the user is new or unverified:
    • Click Add again.
    • A form will appear. Fill in details like:
      • Name
      • Role
      • Company
      • Job Title
      • Country
      • Phone number
    • Click Save.
    • An invite email is sent automatically.
    • The user will appear as Invited until they register and verify their account.

To remove a team member:

• Find the user's name in the Team section.
• Click the Remove icon next to their name.
• Confirm the removal by clicking Yes in the dialog box.

Update Project Settings

To update settings for a specific project:

• Go to the Project Details page.
• Click the menu icon.
• Select Settings.

From here, you can:

• Data Configuration
  • Choose the default export format for your data.
  • Set your preferred delivery options.
  • Click Save to apply changes.
• Preferences
  • Go to the Preferences tab to manage notifications.
  • Enable or disable email and push notifications for:
    • Projects
    • Crawlers
    • Runs
    • Schedules
    • Users
    • System Updates
  • You can enable all or mute all at once if needed.
  • Click Save when done.
• User Permissions
  • From the menu, select User Permissions.
  • Open the Select User dropdown to see all users linked to the project.
  • Pick the user whose permissions you want to adjust.
  • Assign the specific permissions they should have.
  • For crawler access:
    • Select the crawlers from the dropdown list.
  • Click Save to apply changes.