Marown/websegura
1
0
Fork 0
mirror of https://github.com/PucelaBits/websegura.git synced 2024-07-04 12:05:45 +02:00
Code Issues Packages Projects Releases Wiki Activity

chore: configurable crawler settings (#148)

Browse Source
This commit is contained in:
Guido García 2021-02-22 22:00:45 +01:00 committed by GitHub
parent 6d0bb4bf27
commit eedcaccf48
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
1 changed files with 2 additions and 2 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

4
crawler/sites-parser.js
View File

@ -5,9 +5,9 @@
const fs = require("fs");
const glob = require("fast-glob");
const MAX_DAYS_TO_REFRESH = 2; // can be increased when we have many sites to scan
const MAX_DAYS_TO_REFRESH = process.env.CRAWLER_MAX_DAYS_TO_REFRESH || 2; // can be increased when we have many sites to scan
const MAX_TIME_TO_REFRESH_MILLIS = MAX_DAYS_TO_REFRESH * 24 * 60 * 60 * 1000;
const MAX_RESULTS = 200;
const MAX_RESULTS = process.env.CRAWLER_MAX_RESULTS || 200;
/**
* Obtiene los ficheros global, de comunidades y provincias.