# Generate the robots.txt file for a site

`seo-in-nextjs` can generate the `robots.txt` file for your Next.js site with automatic configuration.

## Enabling robots.txt generation

1. Create a `robots.ts` or `robots.js` file in your project's app folder.

2. Import the `robotsTxt` function into the file.

   <Tabs syncKey="lang">

   <TabItem label="TypeScript">

   ```ts
   // app/robots.ts
   export default function robots(): MetadataRoute.Robots {
     return robotsTxt();
   }
   ```

   </TabItem>

   <TabItem label="JavaScript">

   ```js
   // app/robots.js
   export default function robots() {
     return robotsTxt();
   }
   ```

   </TabItem>

   </Tabs>

3. Open http://localhost:3000/robots.txt in your browser to see the robots file.

## Advanced robots configuration

By default, `robotsTxt()` generates a `robots.txt` file with sensible default rules that work well for most cases.

However, you can customize the configuration by passing an options object to the function to extend or override these defaults.

The configuration allows you to specify custom rules, user agents, and crawl delays.

<Tabs syncKey="lang">

<TabItem label="TypeScript">

```ts
// app/robots.ts
export default function robots(): MetadataRoute.Robots {
  return robotsTxt({
    rules: [
      {
        userAgent: "*",
        allow: "/",
        disallow: ["/admin", "/private"],
      },
      {
        userAgent: "Googlebot",
        allow: "/",
        crawlDelay: 10,
      },
    ],
  });
}
```

</TabItem>

<TabItem label="JavaScript">

```js
// app/robots.js
export default function robots() {
  return robotsTxt({
    rules: [
      {
        userAgent: "*",
        allow: "/",
        disallow: ["/admin", "/private"],
      },
      {
        userAgent: "Googlebot",
        allow: "/",
        crawlDelay: 10,
      },
    ],
  });
}
```

</TabItem>

</Tabs>

## Related

View related API references.

[robotsTxt](https://seo-in-nextjs.dlcastillop.com/docs/reference/robots-txt)