To be successful at digital marketing for any kind of website, detailed competitor analysis is essential. For this SEMrush is an ideal way to gain detailed information on many aspects of how a competitor’s website is performing in the search results, telling us how much traffic they receive from organic search, what keywords they rank for along with search volumes and positions, as well as advertising research.

Often we need to analyse several competitors rather than just one; one way of doing this easily and at scale for top-level information is to use the excellent URL Profiler software, which can give you the following information (once a SEMrush API key is provided) simply by adding a list of URLs. It can also obtain a large amount of other information (such as Majestic, Ahrefs and Moz data) which makes it great for large scale domain analysis.

Whilst this is very good for gaining an overview of competitor’s traffic and rankings, sometimes we have to analyse a large number of competitors in more detail. In the below example, we will be obtaining all the keywords for 18 potential competitor websites and combining all of them into one large .csv file. This process takes some time to set up initially but will speed up competitor analysis if the same process is then used in the future.

Analysing competitors in SEMrush

First, we will need a list of competitors (in this example we will be looking at shops selling sunglasses online)

We’ll be using the URL generating spreadsheet that Jacob King designed here. As well as SEMrush there are sections for and Ahrefs already filled out with URL strings, so the below process can be applied to those tools as well.

First, make a copy of the sheet for yourself and trim the URLs to root here. Then, using a URL copying add-on such as Copy All URLs, take all the competitor URLs and put them in the SEMrush section of the spreadsheet, then drag down the URLs in the ‘leave alone’ column to fill it out.

semrush urls

Once the SEMrush-specific URLs are ready, use the add-on again to open them all up. We’ll then need to export the keywords in CSV format from each tab.

semrush urls info

Once all the CSV files are downloaded, create a new folder on your desktop and put all the files in it. Then, create a blank CSV file in the same folder. We’re now going to use the command prompt to combine all of the CSV files together. Open it up and use the following sequence of commands:

cd desktop

cd (combining folder name)

Copy * (combining file name).csv

When it asks what you want to copy, type ‘all’

semrush urls cmd

All of the .csv files should now be in the same file, ready for analysis. Be aware that combining too many large files together may cause Excel to crash!

Some hints for working with the data:

semrush urls csv

It may be a good idea to apply filters to remove the excessive column headers from all of the separate reports to keep the data clean. Speaking of filters, if a particularly large amount of keywords are in the report it may be a good idea to apply some quite aggressive filtering. For example, we could remove branded terms, irrelevant terms, and very low/very high search volumes if necessary to give us a better list of terms to target. We could additionally remove any keywords which have a particularly high cost per click, if we’re doing organic research for developing informational content (on the basis that heavily bidded-on terms in Adwords are likely to be very commercialised.)

Choosing easy/difficult keywords

SEMrush’s ‘keyword difficulty index’ in combination with the ‘competition’ level can be very helpful in determining high/low difficulty keywords more easily, which can be excellent for targeting easy longtail keywords.

Identifying high-performing competitor pages

Also, we can work out which pages of our competitors are performing particularly well in search so we can replicate their content. To do this, we can do an A-Z filter on the ‘URL’ column and look for pages that are ranking for lots of keywords (with Google Hummingbird now more established, this is also a good way of identifying potential topic clusters for your content) additionally, by filtering on traffic %  we can achieve similar results. Just be sure to discount homepages and branded searches which tend to account for most traffic in a lot of cases.

Identifying opportunities from lazy competitors

SEMrush records the previous position for a keyword as well as the current position that the competitor/page held, so as a result by filtering on both search volume, position and previous position (advanced filtering rules may be needed here) it should be possible for us to see where competitors are falling down the rankings – perhaps an old, authority piece of content that hasn’t been updated in a long time, or product pages that have fallen out of favour due to a lack of optimisation.

Finding out overly saturated/less saturated keywords and topics

The ‘number of results’ column provides a great snapshot of how many results there are for each keyword, which can be useful information if we use it correctly. Generally, Google will show far more results for more general queries (even though we would never see them all due to the 1,000 search result limit) and a lot fewer for very specific queries, where the algorithm may decide that far fewer documents are relevant.

Typing ‘SEO’ for example, reveals 512 million results, whereas ‘SEO tips for WordPress’ has far fewer (about 8 million.) By using this sensibly we can work out which overly generic queries to avoid, whilst targeting the more niche, longtail queries that are far less competitive. It still pays to be cautious about search volume, as going after too many low or very low volume keywords can have the end result of only generating only a small trickle of traffic to your site.

About the author:

Ben writes about digital marketing and SEO for Agency51. He particularly enjoys writing about utilising industry tools and software in SEO campaigns



19 − eighteen =