libreCatastro

OpenSource platform for downloading and querying Spanish Official Cadaster Registry (Catastro)

catastral catastro docker elasticsearch elk opendata opensource python registro scrapping spanish-cadaster visualization

Go to file

J af53ec5b4e Adds gitignore		2019-09-23 13:18:21 +02:00
.idea	Adds initialize_elasticsearch script to configurate the ES index	2019-09-16 17:45:24 +02:00
src	Removes debug print	2019-09-23 13:17:37 +02:00
.env	Checks if address already present in ElasticSearch and skips it. Adds ENV var to docker-compose	2019-09-23 13:01:05 +02:00
.gitignore	Adds gitignore	2019-09-23 13:18:21 +02:00
README.md	Refactors and disgregates scrapping and parsing into different classes for maintenability	2019-09-21 15:11:32 +02:00
docker-compose.yml	Checks if address already present in ElasticSearch and skips it. Adds ENV var to docker-compose	2019-09-23 13:01:05 +02:00
initialize_elasticsearch.py	Adds UTF-8 headers to all files	2019-09-20 20:00:36 +02:00
libreCadastro.py	Refactoring of tests. Added health check and some minor changes.	2019-09-22 14:48:41 +02:00
requirements.txt	Adds XML multiparcela. Fixes several bugs.	2019-09-20 19:15:32 +02:00

README.md

#libreCATASTRO An opensource, MIT-licensed application that scraps the official Spanish Cadaster registry and stores information in Elastic Searcher.

Features

Scrapping

From XML webservices. Check http://www.catastro.meh.es/ws/Webservices_Libres.pdf
From HTML
Scraps by zone/site and by property in them
Scraps rural and urban properties
Retrieves a picture of every property

Storing

Stores in ElasticSearch
Allows visualization in Kibana

Visualization

Includes a configured Kibana that shows.

A heatmap in the map of Spain (World) where the properties are
All data in tables
The picture of the property

DoS Warning

Spanish Cadaster has set restrictions, banning temporarily IPs that more than 10 queries in 5 seconds. A sleep command has been set to 5sec where needed, and can be configured at your own risk.

Installation

Having Docker and Docker-compose installed, run first:

docker-compose up -d

Then configure ElasticSearch index:

python3 initialize_elasticsearch.py

That simple!

Execution

python main.py[--coords] [--pictures] [--filenames filename1 filename2 ...] [--provinces province1 province2 ...] [--sleep sleep_time] [--html]