What is a sitemap

Sitemap is an xml file often available at /sitemap.xml path that either links to all pages on your website to gives link to other sitemap urls which might list all those pages. Thsi helps search engines discover pages on your site.

Search engines like Google visit your sitemaps to discover new links published on your website. If you do not have a sitemap Google will still be able to discover them when they crawl your website but not all links could be discoverable in this manner.

<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9" xmlns:news="http://www.google.com/schemas/sitemap-news/0.9" xmlns:xhtml="http://www.w3.org/1999/xhtml" xmlns:mobile="http://www.google.com/schemas/sitemap-mobile/1.0" xmlns:image="http://www.google.com/schemas/sitemap-image/1.1" xmlns:video="http://www.google.com/schemas/sitemap-video/1.1">
<url>
<loc>https://www.tanvi.dev/login</loc>
<lastmod>2023-07-29T22:03:58.888Z</lastmod>
<changefreq>daily</changefreq>
<priority>0.7</priority>
</url>
<url>
<loc>https://www.tanvi.dev/register</loc>
<lastmod>2023-07-29T22:03:58.888Z</lastmod>
<changefreq>daily</changefreq>
<priority>0.7</priority>
</url>
<url>
<loc>https://www.tanvi.dev/privacy</loc>
<lastmod>2023-07-29T22:03:58.888Z</lastmod>
<changefreq>daily</changefreq>
<priority>0.7</priority>
</url>
<url>
<loc>https://www.tanvi.dev/terms</loc>
<lastmod>2023-07-29T22:03:58.888Z</lastmod>
<changefreq>daily</changefreq>
<priority>0.7</priority>
</url>
<url>
<loc>https://www.tanvi.dev/blog/1-tailwind-is-better-than-bootstrap</loc>
<lastmod>2023-07-29T22:03:58.888Z</lastmod>
<changefreq>daily</changefreq>
<priority>0.7</priority>
</url>
<url>
<loc>https://www.tanvi.dev/blog</loc>
<lastmod>2023-07-29T22:03:58.888Z</lastmod>
<changefreq>daily</changefreq>
<priority>0.7</priority>
</url>
<url>
<loc>https://www.tanvi.dev</loc>
<lastmod>2023-07-29T22:03:58.888Z</lastmod>
<changefreq>daily</changefreq>
<priority>0.7</priority>
</url>
</urlset>

Nextjs and Sitemaps

NextJs is a very popular frontend framework these days which is used by react developers to build their frontends. It is capable generating websites statically as well as dynamically. I have written about this in detail in my post building higher performance websites using nexts.

Becuse NextJs can generate pages statically during build time, it can also know the paths during build time and hence it can generate sitemaps very easily. Not all paths can be generated during build time as some of them will be added dynamically in the backend, but these paths can be very easily added to the sitemap during the next build cycle.

Always generate sitemaps in a periodic jobs

Sitemaps are only used by search engines and search engines only occasionally crawl your website depending on how important they think your web pages are. Hence there is no need to generate sitemaps every time there is an update to your site, rather you can generate them at a periodic cadence.

If you write your own code that generates the sitemap dynamically everytime search engine crawler visits /sitemap.xml that code might waste too many compute cycles, also this generation can be slow and hence search engines might think your site suffers from low performance issues.

It makes sense to generate sitemaps once and then cache them until the next generation cycle. It is upto you to decide what cadence to follow but for most, a simpel post build generation would workd just fine.

Using 'next-sitemap' to generate sitemap in NextJs app.

I recommend using "next-sitemap" module to make this process of sitemap generation incredibely simple.

Installation

Install the module using following command.

npm install next-sitemap --save

Please refer to next-sitemap documentation if you want more details.

Configuration

Search engines often check a file /robots.txt before it can crawl your website. It makes sense to add this file as well to your sitemap generation process.

We first add sitemap file and robots.txt file to gitignore becuase they will be generated during build time. Add following lines to your .gitignore file.

public/robots.txt
public/sitemap*
  

Add a file "next-sitemap.config.js" to the root path of your nextjs app.

/** @type {import('next-sitemap').IConfig} */
module.exports = {
  siteUrl: process.env.NEXT_PUBLIC_APP_URL,
  generateRobotsTxt: true, // (optional)
  // ...other options
}
  

After this we add sitemap generation to the post build process.

Modify your package.json file and add the following line under "scripts" object.

 "postbuild": "next-sitemap"
 

An example package.json file will look like this.

{
  "name": "tanvi.dev",
  "version": "0.2.0",
  "private": true,
  "scripts": {
    "start": "next start",
    "lint": "next lint",
    "preview": "next build && next start",
    // This is the line you add. 
    "postbuild": "next-sitemap"
  },
  "dependencies": {
  // Your deps go here. 
  }
  
  

Testing

You can test if the sitemap was generated or not by running the following command.

npm run postbuild

Or

npm run build

Conclusion

Generation of sitemaps in nextjs apps is relatively simple process and should not take more than 10 minutes of your time.