Setting Up Google Indexing

Why Google Indexing API

Google created its Indexing API so that Google Jobs Search can be notified in real time when a job is posted or removed from a job board.

By using this tool, Google gets the most up to date data for the integrity of its Job Listings and you get the benefit of your jobs showing up in Google mere seconds after you post it and/or the job being removed from Google as soon as you remove it.

Full support for the Google Indexing API is now included in both the Pro and All Access versions of Matador Jobs.  It is also available to non-Matador users of WP Job Manager as a separate plugin.

Using it, however, requires some complicated set up. Whether you are using Matador Jobs or Google Indexing API for WP Job Manager, this guide should help you get started.

Warning: Set Up Is Not Easy

We are professional web developers and we have a hard time getting the Google Indexing API set up! The steps are complicated, rely on programmers’ jargon or terms, and are not simple for a non-technical user. But don’t worry! We’ll do our best.

Before You Begin

Here are some things you must have before you attempt to begin this process:

  • The Google logon for the Google Search Console (formerly Webmaster Tools) for the site you are installing Indexing API onto. This is typically also the Google account that owns the Google Analytics for your site. It is common for this Google account to be controlled by the owner of the company and/or the original developer of the website. A delegated user will not work.
  • FTP or SSH access to your site in case you need to add a Google domain verification file during the process, should your domain not already be verified.

Step 1: Get Google API Keys

  1. Log into the Google account that is tied to your sites Google Search Console. 
  2. Go to the Google Services Account page.
  3. If you see a headline for Service Accounts, you may skip to the next step. If you don’t see a headline for Service Accounts, you need to do the following:
    1. Click Create Project.
    2. In the prompt, name the “project” something like “My Job Website”, for example “Matador Jobs Website”.
    3. For the rest of the options in the prompt, generally, the recommended defaults are fine. 
    4. Submit the form and you’ll be redirected to the Service Accounts page.
  4. On the Service Accounts page, click on Create Service AccountNote: if you are familiar with Google APIs and already have a service account for this site, you may use an existing service account. (advanced users only).
  5. In the Create service account window, type a name for the service account.
    1. We recommend using “Google Indexing API for Matador” or something like that.
    2. The ID will be auto-generated.
    3. A description is optional.
  6. You can skip assigning ownership or user access of the API tool.
  7. You are returned to the list of Service Accounts. Select the three dots menu next to the service account you just created and select Create a Key. Follow the prompts to create a JSON key.

A file will be downloaded to your machine. This plain-text file will contain a bunch of gibberish that is actually your JSON-formatted credentials. Save this file in a safe spot. They cannot be regenerated.

This is the only copy of this file you will get, so make sure you don’t lose this. Keep a copy in a shared folder at your business related to your website. You should not add this file into version control for a public repo or a repo that could ever be public.

This file will look something like this (we replaced sensitive data in our file’s screenshot with ellipses):

{
  "type": "service_account",
  "project_id": "long-carving-228615",
  "private_key_id": "...",
  "private_key": "...",
  "client_email": "demo-matador-com@long-carving-228615.iam.gserviceaccount.com",
  "client_id": "...",
  "auth_uri": "https://accounts.google.com/o/oauth2/auth",
  "token_uri": "https://oauth2.googleapis.com/token",
  "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
  "client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/demo-matador-com%40long-carving-228615.iam.gserviceaccount.com"
}

There is one thing you need to take note of in that file, and it is the part “client_email”. In the next few steps, we’ll refer to the client email. The client email is NEVER your personal email. It will always end with .iam.gserviceaccount.com.

Step 2: Add Ownership For Client Email in Webmaster Tools/Search Console

Now that you have keys, you are only halfway there. Just as if you found a random key on the ground, it’s only useful if you know where it goes. In this case, we need to tell Google that the keys you generated are allowed to work on your website.

You need to go to Google Search Console and add the API key’s service account email as an owner for your site.

  1. Go to Google Search Console property users/owners management page.
  2. Click “Add an Owner”
  3. Enter the client_email ended in .iam.gserviceaccount.com into the field. Click continue.

Note: As of the last update to this help document, the instructions in Google’s setup guide (that we based this off of) instructs users to assign ownership in Google Search Console. Unfortunately, for many users, this does not work. Therefore we will also show you how to do this in the legacy Google Webmaster Tools.

  1. Go to Google Webmaster Tools Verification page.
  2. Select the URL for your jobs site. If you do not see it, you will need to go through the process of adding a property, which is out of the scope of this document, but also not hard.
  3. Go to the bottom of the page and click “Add Owner.”
  4. Enter the client_email ended in .iam.gserviceaccount.com into the field. Click continue.

NOTE: the Search Console account URL MUST match your site URL exactly. Check the HTTPS and if www or not! Older sites that were added to Webmaster tools may have URLs like http://your-site.com/ or http://www.your-site.com/. If your site is now https://your-site.com/ you will need to add a new property, unfortunately, and delete the old one.

Step 3: Input Key and “Email” Into Matador/WPJM Settings

Go to your settings screen in either Matador Jobs or WP Job Manager and do the following:

  1. Insert the whole contents of the JSON file into the private key field. Our settings save will extract the part needed. After saving, this will look different, but that is okay, its because the part we needed to keep has been retained!
  2. Into the email section, enter the service email address–the one ending with iam.gserviceaccount.com.
  3. Save settings.

Step 4: Test and Enable API works

Once you have saved the settings after inputting the service account email and API private key, click on the test page link under the email address field.

This will create a Use the Google Test/Debug page. Here, errors are output to the screen. You may see a success message, and that is great! That means you are good to go. If you get an error, skip the next section for tips.

Step 5: Turn On “Send Google Job Changes”

Really, that is it. You’re now on, and indexing immediately you post a job!

Errors

If you get an error during testing, look at the output on the test page for hints. Here are common issues:

  • “error”: “invalid_grant”: This means your service account email address could be wrong. Check the email is from the JSON file it will end with iam.gserviceaccount.com

  • API not active:  This means you created the API project but didn’t enable it. The error message has a URL you can follow to open the project for activation. The link looks like this https://console.developers.google.com/apis/api/indexing.googleapis.com/overview?project=XXXXXXXXX. Once you follow that link, click “enable” and you’ll be good to go.

  • User, not an owner: If your error explains you are a user but not an owner, you did non add the user or make it an OWNER in your Google Search Console. Go back to step 2 and repeat those steps.

Reload the test page as needed to test as you fix the issues.

Additional Considerations

You are limited to 200 changes per day, so if your company regularly creates, updates, or deletes jobs more than 200 times a day, your last changes may not be notified to Google.

While we are working on expanding the feature to include use of Google’s batch processing system–and we’ll be sure to tell you when we get it done–you may want to check out this page on the Indexing API quota and follow the instructions to apply for a greater quota to better fit your business’s volume.

We expect that Google Indexing API pings will cause your pages to be indexed even if you’re in a test mode or your robots.txt is “nofollow”. We recommend that this system should remain off until your site is “live”.