#Screaming frog seo spider 4.1 generator
#Screaming frog seo spider 4.1 software
#Screaming frog seo spider 4.1 code

Use SEMrush to look at your competitor keywords and pages.Step 4.2 - Identify gaps and start content ideation Run a crawl through URL Profiler or look at Search Metrics data for their ranking pages and compare to your list of content.

Use SEMrush to look at your competitors’ keywords and pages.

Complement with WMT data from queries and CTRs to look for opportunities (high impressions and low CTR).

Use Google Analytics’ data and analyse to identify important pages that need attention (high volume low conversion etc).

Step 4 - Identify pages and areas of opportunity

Consider focus of the content audit, as this will set out the rest of the audit intensity.

Links pointing to it (from Majestic, Ahrefs or Web Master Tools).

Format the Excel spread sheet to include all important tabs.

Don’t forget to run the URLs through URL Profiler to get all the additional metrics for the master Excel doc.

Export all URL data from Screaming Frog, Google Analytics and Webmaster Tools.

Crawl the site using your favourite crawler.

Here is a bird’s eye view of the whole process: The content audit is one of the best levers a content marketer can pull, without having to send a huge costing proposal past the board. The content audit of a website is the evaluation of all or part of your website content. Consequently, some of the more simple tasks are overlooked and not fully leveraged. I’ve previously worked in an in-house environment, where it’s easy to get bogged down in the usual routine projects and acquisition opportunities. I know this may seem a bit basic for most marketers out there, but you would be surprised how many times I’ve come across businesses that rely heavily on their organic traffic to generate the majority of their sales, yet have never done a content audit. In particular, the ability to evaluate and leverage onsite assets such as conversion rate optimisation and website content.

#Screaming frog seo spider 4.1 software

Click to share on LinkedIn (Opens in new window)ĭigital marketers are constantly seeking new ways to increase their website traffic, often looking to third party platforms, new emerging software and other off-site tactics, when the solution is within their existing online assets.Click to share on Facebook (Opens in new window).Click to share on Twitter (Opens in new window).If you have a big site you should increase the SEO spider’s memory. Under ‘configuration’ you can also choose to include or exclude crawling of URLs or files (such as images, JS, CSS etc). This helps to avoid information overload first of all, but also speeds up the process and uses less memory. Intelligent Spidering – If you have a site with thousands of pages (you’ll need the premium version ideally!), it’s best to spider the site in sections using the above method.If you have a more complicated set up like subdomains and subfolders you can specify both. Hence, if you only want to crawl the blog of your website which is in its own subfolder/sub directory, simply spider the URI with file path for example –. The tool spiders from directory path onwards.The tool spiders (only) the subdomain you enter and treats other subdomains it encounters as external links.

#Screaming frog seo spider 4.1 generator

XML Sitemap Generator – You can create a basic XML sitemap using the SEO spider.(Please note – This is not a data extraction or scraping feature yet.)

#Screaming frog seo spider 4.1 code

Custom Source Code Search – The spider allows you to find anything you want in the source code of a website! Whether that’s analytics code, specific text, or code etc.

User-Agent Switcher – Crawl as Googlebot, Bingbot, or Yahoo! Slurp.

Images over 100kb, missing alt text, alt text over 100 characters

Images – All URIs with the image link & all images from a given page.

Follow & Nofollow – At link level (true/false).

Outlinks – All pages a URI links out to.

Canonical link element & canonical HTTP headers.

Meta Refresh – Including target page and time delay.

Meta Robots – Index, noindex, follow, nofollow, noarchive, nosnippet, noodp, noydir etc.

H2 – Missing, duplicate, over 70 characters, multiple.

H1 – Missing, duplicate, over 70 characters, multiple.

Meta Keywords – Mainly for reference as it’s only (barely) used by Yahoo.

Meta Description – Missing, duplicate, over 156 characters, multiple.Page Title – Missing, duplicate, over 70 characters, same as h1, multiple.Duplicate Pages – Hash value / MD5checksums lookup for pages with duplicate content.

URI Issues – Non ASCII characters, underscores, uppercase characters, dynamic uris, long over 115 characters.

External Links – All followed links and their subsequent status codes.

Redirects – (3XX, permanent or temporary).

Errors – Client & server errors (No responses, 4XX, 5XX).

A quick summary of some of the data collected.