gulp-robots
v2.0.4 robots txt generator gulp gulpplugin
gulp-robots
Robots.txt generator for Gulp. Simple wrapper around robots-generator. Requires Node 4+. Installed through NPM with:
npm install gulp-robots --save-dev
Check out robots-generator for example options. This module scans your HTML for <link rel="sitemap" />
. Example usage:
var robots = require('gulp-robots');
gulp.task('default', function () {
gulp.src('index.html')
.pipe(robots({
useragent: '*',
allow: ['folder1/', 'folder2/'],
disallow: ['cgi-bin/']
}))
.pipe(gulp.dest('robots.txt'));
});
If you need an ES5 build for legacy purposes, just require the ES5 file:
var robots = require('gulp-robots/es5');
To build the ES5 version, run the following and remember to require the ES5 version.
npm install -g babel-cli
babel --presets es2015 index.js --out-file es5.js