Skip to content

data-sources-setup.md

Latest commit

 

History

History
446 lines (327 loc) · 13.6 KB

data-sources-setup.md

File metadata and controls

446 lines (327 loc) · 13.6 KB

Data Sources Setup Guide

This guide provides step-by-step instructions for setting up the data source integrations used by SEO Machine: Google Analytics 4 (GA4), Google Search Console, DataForSEO, and WordPress.

Table of Contents

  1. Google Analytics 4 (GA4) Setup
  2. Google Search Console Setup
  3. DataForSEO Setup
  4. WordPress Setup
  5. Testing Your Integrations
  6. Troubleshooting

Google Analytics 4 (GA4) Setup

Google Analytics 4 provides traffic data, user behavior metrics, and page performance insights for your website.

Prerequisites

  • Admin access to your Google Analytics 4 property
  • A Google Cloud project (will be created if you don't have one)

Step 1: Enable the Google Analytics Data API

  1. Go to the Google Cloud Console
  2. Select or create a project for your SEO Machine integration
  3. Navigate to APIs & Services > Library
  4. Search for "Google Analytics Data API"
  5. Click on the API and click Enable

Step 2: Create a Service Account

  1. In Google Cloud Console, go to APIs & Services > Credentials
  2. Click Create Credentials and select Service Account
  3. Fill in the service account details:
    • Name: seo-machine-ga4 (or any descriptive name)
    • Description: "Service account for SEO Machine GA4 integration"
  4. Click Create and Continue
  5. Skip the optional "Grant this service account access to project" step (click Continue)
  6. Skip the optional "Grant users access to this service account" step (click Done)

Step 3: Create and Download the Service Account Key

  1. Find your newly created service account in the Credentials page
  2. Click on the service account email
  3. Go to the Keys tab
  4. Click Add Key > Create new key
  5. Select JSON as the key type
  6. Click Create
  7. The JSON key file will automatically download to your computer
  8. Important: Keep this file secure! It provides access to your GA4 data

Step 4: Grant Access to Your GA4 Property

  1. Go to Google Analytics
  2. Select your GA4 property
  3. Click Admin (gear icon in bottom left)
  4. Under Property, click Property access management
  5. Click the + button in the top right and select Add users
  6. Enter the service account email (from Step 2, looks like seo-machine-ga4@your-project.iam.gserviceaccount.com)
  7. Select the role Viewer (read-only access)
  8. Uncheck "Notify new users by email"
  9. Click Add

Step 5: Get Your GA4 Property ID

  1. In Google Analytics, click Admin
  2. Under Property, click Property Settings
  3. Copy your Property ID (format: 123456789)

Step 6: Configure the Integration

  1. Rename your downloaded JSON key file to ga4-credentials.json
  2. Move the file to: credentials/ga4-credentials.json
  3. Create a .env file in the root directory if it doesn't exist
  4. Add the following line to your .env file: 
    GA4_PROPERTY_ID=123456789
    (Replace with your actual Property ID from Step 5)

Google Search Console Setup

Google Search Console provides search query data, click-through rates, average positions, and impressions for your website in Google Search results.

Prerequisites

  • Verified ownership of your website in Google Search Console
  • A Google Cloud project (same one used for GA4 is fine)

Step 1: Enable the Google Search Console API

  1. Go to the Google Cloud Console
  2. Select the same project you used for GA4 (or create a new one)
  3. Navigate to APIs & Services > Library
  4. Search for "Google Search Console API"
  5. Click on the API and click Enable

Step 2: Create a Service Account (or Reuse Existing)

Option A: Reuse Your GA4 Service Account (Recommended)

  • You can use the same service account created for GA4
  • Skip to Step 3 and use the same JSON key file

Option B: Create a New Service Account

  1. In Google Cloud Console, go to APIs & Services > Credentials
  2. Click Create Credentials and select Service Account
  3. Fill in the service account details:
    • Name: seo-machine-gsc (or any descriptive name)
    • Description: "Service account for SEO Machine GSC integration"
  4. Click Create and Continue
  5. Skip the optional steps and click Done
  6. Create and download a JSON key (same process as GA4 Step 3)

Step 3: Grant Access to Your Search Console Property

  1. Go to Google Search Console
  2. Select your property (website)
  3. Click Settings in the left sidebar
  4. Click Users and permissions
  5. Click Add user
  6. Enter the service account email (e.g., seo-machine-ga4@your-project.iam.gserviceaccount.com)
  7. Select permission level: Full (required for API access, but only provides read access)
  8. Click Add

Step 4: Get Your Site URL

Your site URL is the property name in Search Console, typically one of:

  • https://yoursite.com/ (for URL prefix properties)
  • sc-domain:yoursite.com (for domain properties)
  • Check in Search Console settings to confirm the exact format

Step 5: Configure the Integration

Using the Same Service Account as GA4 (Your Current Setup):

Since you're using the same service account for both GA4 and GSC, the configuration is simple:

  1. Credentials file: Use the existing credentials/ga4-credentials.json (no additional file needed)
  2. Add to your .env file: 
    GSC_SITE_URL=https://yoursite.com/
GSC_CREDENTIALS_PATH=credentials/ga4-credentials.json
  3. The integration will automatically use the shared credentials file

If Using a Separate Service Account (Alternative):

  1. Rename the JSON key to gsc-credentials.json
  2. Move to: credentials/gsc-credentials.json
  3. Add to your .env file: 
    GSC_SITE_URL=https://yoursite.com/
GSC_CREDENTIALS_PATH=credentials/gsc-credentials.json

DataForSEO Setup

DataForSEO provides keyword research data, search volume, competition metrics, and SERP analysis.

Prerequisites

  • A DataForSEO account
  • API credits (free trial available, or paid plan)

Step 1: Create a DataForSEO Account

  1. Go to DataForSEO
  2. Click Sign Up or Get Started
  3. Complete the registration process
  4. Verify your email address

Step 2: Get Your API Credentials

  1. Log in to your DataForSEO account
  2. Go to the Dashboard
  3. Navigate to API Access or API Credentials
  4. You'll see your:
    • Login (username/email)
    • Password (API password, different from account password)
  5. Copy both values

Note: DataForSEO uses Basic Authentication with login/password, not API keys.

Step 3: Add Credits to Your Account

  1. In the DataForSEO dashboard, go to Billing or Add Credits
  2. Choose a payment option:
    • Free Trial: Usually provides $1-5 in credits to test
    • Pay As You Go: Add credits as needed
    • Monthly Plan: Subscribe for discounted rates
  3. Complete the payment process

Step 4: Configure the Integration

Add the following to your .env file:

DATAFORSEO_LOGIN=your_username_or_email
DATAFORSEO_PASSWORD=your_api_password

API Usage and Costs

DataForSEO charges per API request. Common pricing:

  • Keywords For Site: ~$0.10 per 1000 keywords
  • Search Volume: ~$0.20 per 1000 keywords
  • SERP Analysis: ~$0.005 per SERP

Tip: Start with the free trial to test the integration, then add credits based on your usage needs.

WordPress Setup

WordPress integration enables publishing articles and landing pages directly from SEO Machine via the REST API.

Prerequisites

  • A WordPress site with REST API enabled
  • Admin access to install plugins
  • Yoast SEO plugin installed (for SEO metadata)

Step 1: Install the MU-Plugin

  1. Copy wordpress/seo-machine-yoast-rest.php to your WordPress site's wp-content/mu-plugins/ directory
  2. This plugin exposes Yoast SEO fields via the REST API for programmatic publishing

Step 2: Create an Application Password

  1. Log in to your WordPress admin
  2. Go to Users > Profile
  3. Scroll to Application Passwords
  4. Enter a name: SEO Machine
  5. Click Add New Application Password
  6. Copy the generated password (you won't see it again)

Step 3: Configure the Integration

Add the following to your .env file:

WP_URL=https://yoursite.com
WP_USERNAME=your_admin_username
WP_APP_PASSWORD=xxxx xxxx xxxx xxxx xxxx xxxx

Step 4: Optional Theme Integration

Add the snippet from wordpress/functions-snippet.php to your theme's functions.php for additional publishing features.

See wordpress/README.md for more details.

Testing Your Integrations

After setting up your data sources, test them to ensure they're working correctly.

Test GA4 Integration

Run the GA4 agent to fetch traffic data:

npm run agent:ga4-traffic-data

Expected output:

  • List of top-performing pages with pageviews
  • Traffic metrics for recent time periods
  • No authentication errors

Test Google Search Console Integration

Run the GSC agent to fetch search data:

npm run agent:gsc-search-data

Expected output:

  • List of top search queries
  • Click and impression data
  • Average position metrics

Test DataForSEO Integration

Run the keyword research agent:

npm run agent:keyword-research "your target keyword"

Expected output:

  • Keyword suggestions related to your query
  • Search volume data
  • Keyword difficulty scores

Troubleshooting

GA4 Issues

Error: "Permission denied"

  • Verify the service account email is added to your GA4 property with Viewer permissions
  • Double-check you're using the correct Property ID in .env

Error: "API not enabled"

  • Make sure you enabled the Google Analytics Data API in Google Cloud Console
  • Wait a few minutes after enabling for changes to propagate

Error: "Invalid credentials"

  • Verify the JSON key file is in the correct location
  • Ensure the file hasn't been corrupted or modified
  • Check that the service account hasn't been deleted

Google Search Console Issues

Error: "User does not have sufficient permissions"

  • Verify the service account is added to Search Console with Full permissions
  • Make sure you're using the correct site URL format in .env

Error: "Site not found"

  • Check the GSC_SITE_URL format matches exactly what's in Search Console
  • Try both formats: https://yoursite.com/ and sc-domain:yoursite.com

DataForSEO Issues

Error: "Authentication failed"

  • Verify your login/password in .env are correct
  • Make sure you're using the API password, not your account password
  • Check for any extra spaces in the credentials

Error: "Insufficient credits"

  • Check your account balance at app.dataforseo.com
  • Add more credits or upgrade your plan

Error: "Rate limit exceeded"

  • DataForSEO has rate limits; wait a few minutes before retrying
  • Consider spreading out API calls over time

General Issues

Environment Variables Not Loading

  • Make sure your .env file is in the root directory
  • Verify there are no syntax errors in the .env file
  • Restart your application after modifying .env

Credentials File Not Found

  • Check the file path is correct: credentials/
  • Create the credentials/ directory if it doesn't exist: 
    mkdir -p credentials/
  • Verify file names match exactly (case-sensitive)

Security Best Practices

Protect Your Credentials

  1. Never commit credentials to Git

    • Add to .gitignore: 
      credentials/
.env

  2. Restrict file permissions:

    chmod 600 credentials/*.json
chmod 600 .env

  3. Rotate credentials regularly

    • Regenerate service account keys every 90 days
    • Update DataForSEO password periodically

  4. Use principle of least privilege

    • Only grant Viewer/Read access where possible
    • Don't share credentials between projects unnecessarily

Backup Your Credentials

Store encrypted backups of your credentials in a secure location:

  • Password manager (1Password, LastPass, etc.)
  • Encrypted cloud storage
  • Secure company vault

Quick Reference

File Structure

seomachine/
├── .env                          # Environment variables
├── credentials/
│   ├── ga4-credentials.json     # GA4 service account key
│   └── gsc-credentials.json     # GSC service account key (or reuse GA4)
├── config/
│   └── competitors.json         # Competitor configuration
└── data-sources-setup.md        # This file

Environment Variables

# Google Analytics 4
GA4_PROPERTY_ID=123456789

# Google Search Console
GSC_SITE_URL=https://yoursite.com/
COMPANY_NAME=Your Company

# DataForSEO
DATAFORSEO_LOGIN=your_username
DATAFORSEO_PASSWORD=your_api_password

# WordPress (optional, for /publish-draft)
WP_URL=https://yoursite.com
WP_USERNAME=your_username
WP_APP_PASSWORD=your_application_password

Useful Links

Need Help?

If you encounter issues not covered in this guide:

  1. Check the official documentation for each service
  2. Review error logs for specific error messages
  3. Verify all prerequisites are met
  4. Test each integration independently to isolate the issue

Last Updated: January 2025