Marown/yacy_search_server
1
0
Fork 0
mirror of https://github.com/yacy/yacy_search_server.git synced 2024-09-21 00:00:13 +02:00
Code Issues Actions Packages Projects Releases Wiki Activity
35b1bd66cd
yacy_search_server/source/de/anomic/plasma
History
orbiter a31b9097a4 preparations for mass remote crawls: 
two main changes must be implemented to enable mass remote crawls:
- shift control of robots.txt to crawl queue (away from stacker). This is necessary since remote
  crawls can contain unchecked urls. Each peer must check the robots to prevent that it is misused
  as crawl agent for unwanted file retrieval
- implement new index files that control double-check of remotely crawled urls

After removal of robots.txt checking from stacker threads, the multi-threading of this process is void.
Multithreading has been removed. Also the thread pools for the crawl threads had been removed, since
creation of these threads is not resource-consuming, for a detailed explanation see svn 4106

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4181 6c8d7289-2bf4-0310-a012-ef5d649a1542
2007-10-29 01:43:20 +00:00
..
cache preparations for mass remote crawls: 2007-10-29 01:43:20 +00:00
crawler preparations for mass remote crawls: 2007-10-29 01:43:20 +00:00
dbImport preparations for mass remote crawls: 2007-10-29 01:43:20 +00:00
parser Add external lib FontBox which is part of the PDFBox (they extracted the font handling code into this package in 0.7.3). 2007-10-18 19:53:52 +00:00
urlPattern clear caches after changing blacklists, closes http://forum.yacy-websuche.de/viewtopic.php?f=6&t=241&p=1964#p1964 2007-09-10 08:15:25 +00:00
plasmaCondenser.java - Changed yFormatter handling to be more flexible and produce more readable code for server pages. There are serverObject.putNum() methods to allow adding of number type values in a formatted form, and put() methods for number types that add them without formatting. This reduces the need to transform them into Strings in server pages and removes the HTML encoding step which is unecessary for numbers. 2007-10-19 04:13:46 +00:00
plasmaCrawlBalancer.java distinct keepOrder strategy, more discrete implementation of enhancement introduced in SVN 4158 2007-10-22 15:26:47 +00:00
plasmaCrawlEntry.java preparations for mass remote crawls: 2007-10-29 01:43:20 +00:00
plasmaCrawlEURL.java fixed problem with redirection: redirected URLs had not been tested with the double-check 2007-10-02 22:40:53 +00:00
plasmaCrawlLoaderMessage.java joined anomic.net.URL, plasmaURL and url hash computation: 2007-09-05 09:01:35 +00:00
plasmaCrawlLURL.java distinct keepOrder strategy, more discrete implementation of enhancement introduced in SVN 4158 2007-10-22 15:26:47 +00:00
plasmaCrawlNURL.java - redesigned crawl start menu, integrated monitoring pages 2007-09-28 01:21:31 +00:00
plasmaCrawlProfile.java - redesigned crawl start menu, integrated monitoring pages 2007-09-28 01:21:31 +00:00
plasmaCrawlRobotsTxt.java
plasmaCrawlStacker.java preparations for mass remote crawls: 2007-10-29 01:43:20 +00:00
plasmaCrawlZURL.java preparations for mass remote crawls: 2007-10-29 01:43:20 +00:00
plasmaDHTChunk.java re-designed some parts of DHT position calculation (effect is the same as before) 2007-10-01 12:30:23 +00:00
plasmaDHTFlush.java
plasmaDHTTransfer.java fix for peer ping problem (implemented a 3-time re-ping); cause for 'Connection reset' still unknown 2007-09-12 00:42:53 +00:00
plasmaGrafics.java some additions and a small bugfix to SVN 4158 2007-10-21 23:26:22 +00:00
plasmaHTCache.java preparations for mass remote crawls: 2007-10-29 01:43:20 +00:00
plasmaParser.java - redesigned crawl start menu, integrated monitoring pages 2007-09-28 01:21:31 +00:00
plasmaParserConfig.java joined anomic.net.URL, plasmaURL and url hash computation: 2007-09-05 09:01:35 +00:00
plasmaParserDocument.java joined anomic.net.URL, plasmaURL and url hash computation: 2007-09-05 09:01:35 +00:00
plasmaRankingCRProcess.java
plasmaRankingDistribution.java re-designed some parts of DHT position calculation (effect is the same as before) 2007-10-01 12:30:23 +00:00
plasmaRankingRCIEvaluation.java joined anomic.net.URL, plasmaURL and url hash computation: 2007-09-05 09:01:35 +00:00
plasmaSearchContainer.java re-implemented post-ranking of search results 2007-09-08 11:50:19 +00:00
plasmaSearchEvent.java fix ArrayIndexOutOfBoundsException 2007-10-04 08:50:33 +00:00
plasmaSearchImages.java joined anomic.net.URL, plasmaURL and url hash computation: 2007-09-05 09:01:35 +00:00
plasmaSearchPreOrder.java joined anomic.net.URL, plasmaURL and url hash computation: 2007-09-05 09:01:35 +00:00
plasmaSearchProcessing.java joined anomic.net.URL, plasmaURL and url hash computation: 2007-09-05 09:01:35 +00:00
plasmaSearchQuery.java steps to enhance remote search performance: 2007-09-26 10:11:50 +00:00
plasmaSearchRankingProfile.java re-implemented post-ranking of search results 2007-09-08 11:50:19 +00:00
plasmaSnippetCache.java preparations for mass remote crawls: 2007-10-29 01:43:20 +00:00
plasmaStore.java
plasmaSwitchboard.java preparations for mass remote crawls: 2007-10-29 01:43:20 +00:00
plasmaSwitchboardQueue.java preparations for mass remote crawls: 2007-10-29 01:43:20 +00:00
plasmaWebStructure.java joined anomic.net.URL, plasmaURL and url hash computation: 2007-09-05 09:01:35 +00:00
plasmaWordConnotation.java - added SSI parser and include directive for <!--# include virtual="<file>" --> 2007-06-26 14:37:10 +00:00
plasmaWordIndex.java Avoid ConcurrentModificationException when accessing the PerformanceQueues page while yacy is indexing. 2007-10-19 23:36:40 +00:00