Commit Graph

13 Commits

Author SHA1 Message Date
orbiter
1689030ee8 refactoring: moved all crawler classes into their own package
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4768 6c8d7289-2bf4-0310-a012-ef5d649a1542
2008-05-06 00:32:41 +00:00
orbiter
e024e3b9cf added new default profiles to distinguish snippet fetch for local and global search
the difference is, that a local search will no not cause a re-indexing of loaded pages

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4731 6c8d7289-2bf4-0310-a012-ef5d649a1542
2008-04-24 08:42:08 +00:00
orbiter
541b817502 refactoring of switchboard queueing
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4591 6c8d7289-2bf4-0310-a012-ef5d649a1542
2008-03-22 01:28:37 +00:00
orbiter
9d693ee635 more generics
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4415 6c8d7289-2bf4-0310-a012-ef5d649a1542
2008-01-29 16:41:09 +00:00
orbiter
842308ea97 - redesigned crawl start menu, integrated monitoring pages
- removed web structure picture from indexing menu and grouped it together with htcache monitor
- added a database for terminated crawls, when a crawl is finished it is automatically moved to the new database
- extended crawl profile edit servlet, shows now also terminated crawls
- option that was used to delete profiles is now redesigned to a function that moves the current crawl to the terminated crawls and removes all urls from the current queues!
- fixed here and there problems with indexing queues
- enhances indexing speed by changing cache flush sizes.
- changed behaviour of crawl result servlet: the list of crawled urls is shown if there is one, othevise the overview window is shown

attention: the new profile databases are not compatible with the old one. current crawls will be lost! the web index is not touched.
next steps: the database of terminated crawls can be used to start with them a new crawl. This is useful if one wants to re-crawl specific pages and wants to use a old crawl profile.


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4113 6c8d7289-2bf4-0310-a012-ef5d649a1542
2007-09-28 01:21:31 +00:00
orbiter
e332b844b2 - enhanced remote search: during waiting time for remote crawls
some urls are fetched so the url cache can be filled with these urls
- the url-prefetch is used to sort out some unresolved urls
- the snippet-fetcher is triggered with the search event id. This is used
  to remove missing snippets from the search cache so they will not be displayed again


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4060 6c8d7289-2bf4-0310-a012-ef5d649a1542
2007-08-26 18:18:35 +00:00
orbiter
5fd1d5a58e added time configuration
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3853 6c8d7289-2bf4-0310-a012-ef5d649a1542
2007-06-10 20:25:33 +00:00
orbiter
e07458bad4 added time-out function to web analysis
the default time-out is 1 second

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3852 6c8d7289-2bf4-0310-a012-ef5d649a1542
2007-06-10 20:00:44 +00:00
allo
5cae5e79c2 nice +/- buttons
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3823 6c8d7289-2bf4-0310-a012-ef5d649a1542
2007-06-07 23:11:13 +00:00
orbiter
8f00c1e896 restrict number of nodes in web structure picture
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3822 6c8d7289-2bf4-0310-a012-ef5d649a1542
2007-06-07 22:56:42 +00:00
orbiter
7fdf31093b automatic focus on one of the crawl starts when web structure is shown
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3811 6c8d7289-2bf4-0310-a012-ef5d649a1542
2007-06-06 20:18:39 +00:00
orbiter
0ae6664ad8 enhanced web structure picture
- hand-over of get properties from web front-end to graphics generation
- added depth-control buttons
- added marking of anchor-points to highlight relation order
- enhanced ymage graphics library

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3757 6c8d7289-2bf4-0310-a012-ef5d649a1542
2007-05-24 15:28:03 +00:00
orbiter
a585b4d41b added web structure image
see http://localhost:8080/WatchWebStructure_p.html

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3747 6c8d7289-2bf4-0310-a012-ef5d649a1542
2007-05-22 15:20:50 +00:00