yacy_search_server/source/net/yacy/crawler/data
Michael Peter Christen 8df8ffbb6d enhanced the snapshot functionality:
- snapshots can now also be xml files which are extracted from the solr
index and stored as individual xml files in the snapshot directory along
the pdf and jpg images
- a transaction layer was placed above of the snapshot directory to
distinguish snapshots into 'inventory' and 'archive'. This may be used
to do transactions of index fragments using archived solr search results
between peers. This is currently unfinished, we need a protocol to move
snapshots from inventory to archive
- the SNAPSHOT directory was renamed to snapshot and contains now two
snapshot subdirectories: inventory and archive
- snapshots may now be generated by everyone, not only such peers
running on a server with xkhtml2pdf installed. The expert crawl starts
provides the option for snapshots to everyone. PDF snapshots are now
optional and the option is only shown if xkhtml2pdf is installed.
- the snapshot api now provides the request for historised xml files,
i.e. call:
http://localhost:8090/api/snapshot.xml?urlhash=Q3dQopFh1hyQ
The result of such xml files is identical with solr search results with
only one hit.
The pdf generation has been moved from the http loading process to the
solr document storage process. This may slow down the process a lot and
a different version of the process may be needed.
2014-12-09 16:20:34 +01:00
..
Cache.java more IPv6 bugfixes 2014-10-06 17:44:27 +02:00
CrawlProfile.java enhanced the snapshot functionality: 2014-12-09 16:20:34 +01:00
CrawlQueues.java ViewFile servlet: update index if newer, 2014-12-05 01:13:37 +01:00
Latency.java - added a new Crawler Balancer: HostBalancer and HostQueues: 2014-04-16 21:34:28 +02:00
NoticedURL.java reduce number of calls to queue.size() because that may be a bottleneck 2014-11-23 20:09:32 +01:00
ResultImages.java fix for image alt attachment to AnchorURLs in html parser. 2014-08-01 12:04:15 +02:00
ResultURLs.java
Snapshots.java enhanced the snapshot functionality: 2014-12-09 16:20:34 +01:00
Transactions.java enhanced the snapshot functionality: 2014-12-09 16:20:34 +01:00