Marown/yacy_search_server
1
0
Fork 0
mirror of https://github.com/yacy/yacy_search_server.git synced 2024-09-19 00:01:41 +02:00
Code Issues Actions Packages Projects Releases Wiki Activity
5,558 Commits 1 Branch 20 Tags 836 MiB
4cf8b08eec
Commit Graph

2527 Commits

Author SHA1 Message Date
orbiter
299189f1a9 added the API icon to the bookmarks, the network page and the search page 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5455 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-01-07 23:45:20 +00:00
orbiter
a1bf687b3b added first API tooltip! 
- description of JSON search result in interactive search

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5454 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-01-07 22:57:32 +00:00
lotus
0d1bd78674 * full site: syntax support e.g. site:de.wikipedia.org 
possible if dots in query would work yet

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5453 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-01-07 21:05:07 +00:00
orbiter
9bed4de280 fix for the search bug introduced in SVN 5449 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5451 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-01-06 23:16:10 +00:00
lotus
d6a5c98080 api banner concept draft 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5450 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-01-06 20:51:22 +00:00
orbiter
b2b7edae18 fixed interactive search 
- added dummy servlet class, because otherwise the template engine is not triggered.
thats so because the yacy httpd works much faster as normal file server without a scan
of the served pages. Therefore each page with templates must now have a class file associated to it.
- fixed json output format of yacysearch

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5449 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-01-06 20:04:09 +00:00
orbiter
c6880ce28b removed the permanent cache flush and replaced it with a periodic cache flush 
The cache is now flushed only for one second every ten seconds. During a crawl the cache
fills up completely, and is only flushed if space is needed for more documents.

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5446 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-01-06 13:51:59 +00:00
lotus
ca80930892 accept leading dots on filetype: and site: search 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5444 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-01-06 10:04:24 +00:00
orbiter
6c7e83909b - refactoring of data access methods to be prepared for new cell data structure 
- removed a memory overhead in collections which prevent OOM Exception in low memory configurations

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5443 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-01-06 09:38:08 +00:00
low012
1af728ae09 *) regex for site operator changed as proposed by Lotus 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5441 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-01-05 18:30:34 +00:00
low012
9e58ae036d *) added site operator which can be used to only show results from a certain domain. example: "test site:edu" shows only documents which contain the word test and which come from an edu domain 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5439 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-01-04 14:58:32 +00:00
low012
19e7c56f7f *) apply filter to dir list to only show .black files as blacklists 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5438 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-01-04 10:14:19 +00:00
orbiter
c4c4c223b9 fixed a problem with attribute flags on RWI entries that prevented proper selection of index-of constraint 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5437 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-01-04 02:27:29 +00:00
low012
4bffe664ca *) moved entry field for new expressions to top of the list as requested in forum (http://forum.yacy-websuche.de/viewtopic.php?f=9&t=1678) 
*) added some Javascript to disable list selection on bottom of list in cases it is not needed (edit, delete) and only enable it if needed (move), if JS is turned off everything will work as usual

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5435 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-01-03 10:18:48 +00:00
low012
9d5d30f877 *) http://forum.yacy-websuche.de/viewtopic.php?f=6&t=1672 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5422 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-12-31 16:50:10 +00:00
orbiter
5448aad328 removed unused code 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5421 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-12-30 12:12:00 +00:00
orbiter
28d2d28573 added support for filetype search 
(just use filetype:<type> in the search query)

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5418 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-12-29 17:57:04 +00:00
orbiter
78c568331e added test channel to /xml/feed.rss 
can be obtained with 
http://localhost:8080/xml/feed.rss?set=TEST
returns always a single feed entry with a fresh date

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5416 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-12-29 12:39:07 +00:00
orbiter
e004da48d3 - added fast fingerprint computation for files (any). Will be used in new index dump method 
- refactoring

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5415 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-12-29 12:22:13 +00:00
low012
eab72424df *) Fixed small bug: When adding new elements to blacklist via import, the blacklist which the elements were added to was supposed to be displayed, which did not work correctly. 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5414 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-12-28 09:58:02 +00:00
low012
0e56675596 *) cleaning up ;-) 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5413 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-12-27 20:09:36 +00:00
low012
cf69557ea2 *) blacklists can be exported as XML or plain text now 
*) blacklist import via file upload works now

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5412 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-12-27 15:38:20 +00:00
low012
1594a15be9 *) explicit mentioning of blacklist in blacklist cleaner 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5411 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-12-27 13:06:05 +00:00
low012
5a89266598 *) new parameters for future use (better blacklist handling for im- and export) 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5403 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-12-19 19:33:08 +00:00
orbiter
e34ac22fbd - added new monitoring servlet at 
http://localhost:8080/PerformanceConcurrency_p.html
- used the new monitoring to do some fine-tuning of the indexing queue

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5402 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-12-19 15:26:01 +00:00
orbiter
d376d81fc4 replaced busy thread control of crawl stacker by blocking threads 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5400 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-12-18 23:18:34 +00:00
orbiter
7535fd7447 - refactoring of CrawlEntry and CrawlStacker 
- introduced blocking queues in CrawlStacker to make it ready for concurrency
- added a second busy thread for the CrawlStacker
The CrawlStacker is multithreaded. It shall be transformed into a BlockingThread in another step.
The concurrency of the stacker will hopefully solve some problems with cases where DNS blocks.

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5395 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-12-17 22:53:06 +00:00
lotus
6569cbbec1 npe fix: http://forum.yacy-websuche.de/viewtopic.php?t=1646 
(break to avoid bad side effects)

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5394 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-12-16 20:53:31 +00:00
orbiter
2802138787 - refactoring of CrawlStacker (to prepare it for new multi-Threading to remove DNS lookup bottleneck) 
- fix of shallBeOwnWord target computation heuristic


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5392 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-12-15 00:02:58 +00:00
lotus
b1e211b258 no error-alert: http://forum.yacy-websuche.de/viewtopic.php?t=1639 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5391 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-12-13 12:04:08 +00:00
orbiter
13cb0916ee changes to statistics and content of thread dump servlet 
(points now more directly to performance leaks without mentioning class calls inside of sun/java calls that cannot be changed anyway)

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5390 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-12-11 20:13:14 +00:00
orbiter
e1acdb952c fix for problem with userDB and bookmarksDB which was caused by changes in kelondroRA in SVN 5376 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5385 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-12-08 00:17:45 +00:00
lotus
e918d64c23 show hand-cursor an labels 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5383 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-12-06 17:32:53 +00:00
orbiter
4a2dac659e more speed hacks: 
- modified and activated write buffer
- increased cache flush factor
- fixed a problem with deadlocking of indexing process

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5382 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-12-05 13:55:48 +00:00
lotus
1fb518a5b4 display <String> etc. 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5380 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-12-04 20:21:53 +00:00
orbiter
47292e696a more performance hacks 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5379 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-12-04 12:54:16 +00:00
orbiter
bd1dc9cd5d thread dump with statistics, a little bit of profiling 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5377 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-12-03 23:26:25 +00:00
orbiter
d39d420b39 performance hacks 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5376 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-12-03 15:38:29 +00:00
lotus
5280ad638d added basic performance page 
other performance settings can be found on advanced settings

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5375 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-12-03 14:10:01 +00:00
lotus
1a51d9fcfd display proper values 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5374 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-12-02 17:57:30 +00:00
orbiter
0b4808ba3d added new interactive search feature: 
- during the user types search queries, the local database is searched
- results are presented interactively

This was implemented using a new JSON result format for search results in YaCy
- added JSON as file format for servlets
- refactoring of current search servlets (xml and html)
- added JSON output format for search results
- added AJAX-based search page, that uses the yacysearch.json selrvlet to print results as a query is typed

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5373 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-12-02 15:24:25 +00:00
lotus
fea82b54ef more contrast on search snippets 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5370 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-11-26 19:57:13 +00:00
lotus
1951d30a62 addendum to last commit 
handle words with length < 3 correctly

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5369 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-11-26 19:43:40 +00:00
lotus
325ba7bfb8 only query words with length > 2 
this is not complete, yet

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5368 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-11-26 16:41:38 +00:00
lotus
489edb4473 improved pattern selection 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5367 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-11-26 10:06:38 +00:00
low012
e423fa9846 *) added method to only get file names in directory listing which match a filter 
*) only files which end with .black will be listed as blacklists
*) added a little bit of Javadoc

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5366 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-11-25 20:26:06 +00:00
lotus
577b53aee6 added more search engines 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5365 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-11-24 13:05:20 +00:00
lotus
7f4d411c0d npe-fix 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5364 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-11-24 13:04:57 +00:00
lotus
1545e5440a * index deletion: checkbox-confirmation 
* watch crawler: less load on exhausted peers; wait for data before reloading again

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5359 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-11-23 12:02:58 +00:00
orbiter
10f5ec1040 reverted last commit (more testing needed) 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5356 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-11-22 00:12:50 +00:00
daburna
ba5b274b8c #translation update: 
-blacklist
-crawlstart
...

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5353 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-11-21 16:45:45 +00:00
orbiter
0ca4bc7b79 - added reader and visualization for mediawiki-export files: 
files exported from mediawiki using the xml schema according to
http://www.mediawiki.org/xml/export-0.3/
can be processed to be viewed in a YaCy servlet.
To acces such a file, place it into
DATA/HTCACHE/mediawiki/
i.e. the export from german wikipedia would be:
DATA/HTCACHE/mediawiki/wikipedia.de.xml
This file can then be accessed using the URL
http://localhost:8080/mediawiki_p.html?dump=wikipedia.de.xml&title=YaCy
if this is done the first time, an index file is created
(for this case: more than 4 million lines must be written, this takes about 15 minutes)
Then try the same url again.

- enhanced also the md5 computation speed


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5352 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-11-20 18:31:52 +00:00
lotus
4f996a7651 fix for logparser pattern 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5349 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-11-17 16:23:17 +00:00
orbiter
867d0f2f56 removed some unnecessary pause delays 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5346 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-11-14 23:36:33 +00:00
lotus
fd83e59f8e new remote search average 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5343 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-11-14 11:50:46 +00:00
orbiter
dba7ef5144 extended crawling constraints: 
- removed never-used secondary crawl depth
- added a must-not-match filter that can be used to exclude urls from a crawl
- added stub for crawl tags which will be used to identify search results that had been produced from specific crawls
please update the yacybar: replace property name 'crawlFilter' with 'mustmatch'.
Additionally, a new parameter named 'mustnotmatch' can be used, which should be by default the empty sring (match-never)

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5342 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-11-14 09:58:56 +00:00
orbiter
0ae84f4f8e set some default values for a crawl start that should cause less confusion and mistakes 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5334 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-11-12 19:48:22 +00:00
lotus
4745e89451 auto-choose crawl type 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5331 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-11-12 14:44:23 +00:00
low012
421d056550 *) changed layout of blacklist adminstration (less cluttered) 
*) it is possible to move/edit/delete more than one entry at a time now
*) it is easier to choose a target for blacklist import now
*) fixed several bugs
*) to be continued...

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5330 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-11-12 00:47:54 +00:00
orbiter
674ad2d55b different handling of error cases that occur during loading files with http or ftp: 
methods throw exception instead of returning an error string

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5328 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-11-11 21:33:40 +00:00
f1ori
ae80f3e6a5 * extend opensearchdescription to support compare_yacy.html 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5324 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-11-09 00:23:19 +00:00
orbiter
1b18d4bcf3 enhancement to crawling and remote crawling: 
- for redirector and  remote crawling place crawling url on notice queue instead of direct enqueueing in crawler queue
- when a request to a remote crawl provider fails, remove the peer from the network to prevent that the url fetcher gets stuck another time again

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5320 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-11-06 12:30:55 +00:00
orbiter
3f746be5d4 - consolidation and refactoring of many DHT target - computing methods 
- implemented vertical DHT acceptance ("my own DHT") to accept new targets
- added new target computation for global search: addresses vertical targets also
- enhanced remote crawling: collection of remote crawl urls if queue has less than 100 entries (was: 0 entries)
- better performance value computations for PPM selection in network configuration

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5319 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-11-06 10:07:53 +00:00
orbiter
d014b2728a Design-check, Extension and Refactoring of DHT target position computation: 
- two different computations (but mathematical equivalent) of the DHT distance had been consolidated
- moved from 0.0 .. 1.0 double-range position computation to 0 .. Long.Max range for DHT targets
- added fast Long - to - hash computation
- high-precision target computation of gaps for new peers
- added new target computation for horizontal and vertical DHT targets (not yet in use)
- old horizontal-only DHT targets will be upwards compatible to new horizontal and vertical DHT positions

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5318 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-11-03 00:27:23 +00:00
low012
83967f8c77 *) servlet does not forget chosen blacklist anymore when editing, moving or delting an entry 
*) move or edit will only be performed if new value actually differs from old one

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5309 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-10-30 00:03:14 +00:00
low012
04e41a392f *) fixed bug where RegExes were not deleted and even added to the list a second time when the user tried to edit them 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5308 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-10-29 22:49:44 +00:00
low012
7bac4796d2 *) added servlet which returns all shared blacklists of a peer without information about which part of YaCy (crawler, proxy, ...) blacklist is activated for (to be used for better online import) 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5306 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-10-27 17:33:43 +00:00
low012
baae3d91b1 *) fixed warning when compiling listManager 
*) fixed display of values of information for which part of YaCy (crawler, proxy, ...) blacklist is activated for
*) replaced regular put() with putXML() in several cases

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5305 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-10-27 16:56:19 +00:00
low012
444575e33d *) prevent XSS when importing blacklist 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5304 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-10-27 11:06:38 +00:00
low012
a99a629ed4 *) quick fix to prevent comments for blog entries which don't exist (http://forum.yacy-websuche.de/viewtopic.php?f=5&t=1554) 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5302 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-10-25 12:04:10 +00:00
low012
00e27e5050 *) fixed bug which made it possible to write files outside of the DATA/LIST directory when creating a new blacklist 
*) a blacklist will only be created if no blacklist with same name exists (some refactoring has been necessary for this)
*) further minor fixes
*) to be continued...

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5301 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-10-25 00:11:03 +00:00
orbiter
0edec2b760 FULL redesign of algorithms in htmlTools to encode/decode strings from/to unicode and html. 
The old process used a not really efficient way to detect html encoding strings in texts.
All calling methods had been adoped to call the new class in an enhanced way with less parameters.

Many classes in interfaces used a XML encoding only (instead of full html conversion from unicode to html); this behavior was not changed with this commit but should be controlled again since it points out possible XSS leaks

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5295 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-10-22 18:59:04 +00:00
orbiter
47f0c3b002 replaced the cacheAdmin with the ViewFile servlet, because the cacheAdmin was an interface to the old HTCACHE data structure which does not exist any more. Changed links to point to the ViewFile servlets. 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5289 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-10-21 11:27:50 +00:00
orbiter
1778fb420d - added some performance tweaks to the new BLOB buffer 
- removed the now superfluous HT storage thread
- reduced number of file decompression by shifting the compression moment to the future


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5286 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-10-19 18:10:42 +00:00
low012
77e41da7d2 *) further propagation of display value (see http://forum.yacy-websuche.de/viewtopic.php?f=5&t=1536) 
*) removed another depreciated parameter "time" which led to ugly -UNRESOLVED_PATTERN- in URL

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5285 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-10-18 19:39:46 +00:00
low012
ff46ce8520 *) fixed display=2 (see http://forum.yacy-websuche.de/viewtopic.php?f=5&t=1536) 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5283 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-10-17 19:57:39 +00:00
orbiter
826ca79735 refactoring and new architecture to store the files of the web cache: 
- files are not stored any more as individual files
- a new database structure using BLOBHeap files stores many cache entries in common files
- all file-writing procedures had been migrated to generate byte[] objects which are written with the new database methods

this is only an intermediate step to the final architecture, where cached files are written together with their metadata in one single database structure.

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5276 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-10-16 21:24:09 +00:00
low012
2b18a9b2c4 *) removed depreciated parameter "time" which led to ugly -UNRESOLVED_PATTERN- in URL 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5275 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-10-16 19:31:29 +00:00
orbiter
7860d5d632 fix for bug in seed list management (cause was bad class overloading, only visual effects!) 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5265 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-10-12 19:51:53 +00:00
lotus
603282bcf4 fix for out of bounds exception 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5264 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-10-11 07:47:34 +00:00
orbiter
ffed5fc415 fixed problem with lost peers in database 
migrated seedDB from BLOBTree to BLOBHeap

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5263 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-10-10 14:40:02 +00:00
lotus
736dd86193 - option enableSimpleConfig can disable hidden tables 
- corrected some Xmx values
- friendlier welcome message format

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5259 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-10-09 12:48:43 +00:00
orbiter
ff68f394dd fix for problem with balancer and lost crawl profiles: 
if crawl profile ist lost, no robots.txt is loaded any more

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5258 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-10-08 18:26:36 +00:00
apfelmaennchen
3717d2057a YaCy-UI: fix for http://forum.yacy-websuche.de/viewtopic.php?f=5&t=1483 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5251 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-10-05 18:50:43 +00:00
orbiter
d0bdcdd57c small changes to attributes of DoS attack protection parameters 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5246 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-10-03 19:44:42 +00:00
orbiter
9ac16f565b - fixed several bugs in database management functions 
- fixed a display bug for the performance graph
- fixed deadlock when initialization of awt happens simultanously
- removed some debugging output

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5245 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-10-03 18:57:02 +00:00
lotus
7fdf65339d system status dropped into next line if seed server was enabled. display needs about 230px, set fixed width again. 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5243 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-10-03 11:42:24 +00:00
lotus
7782a43060 fix if LANGUAGE: was not defined and the end of the query 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5242 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-10-03 11:36:17 +00:00
apfelmaennchen
2c23e6ad34 YaCy-UI: added some features for Admin Console 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5241 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-10-03 05:32:31 +00:00
apfelmaennchen
efcae14714 YaCy-UI: 
- added 'Open' button to search result toolbar
- lets you open all selected search results in new window/tab
- added 'any language' filter as default

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5240 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-10-02 17:38:03 +00:00
lotus
902a0d0f38 fieldset of system status was bigger than defined space. IE overlapped some text. 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5238 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-10-01 20:33:32 +00:00
orbiter
820a03f9d6 - removed some warnings 
- used fix in SVN 5233 for ysearch.java and search.java

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5237 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-10-01 20:20:39 +00:00
lotus
69925a7e91 fix for http://forum.yacy-websuche.de/viewtopic.php?f=6&t=1441 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5236 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-10-01 20:16:50 +00:00
lotus
fe2792e9ce use accept-language header instead of user agent for language detection 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5235 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-10-01 17:47:11 +00:00
lotus
e5904e6a21 removed color definition for input elements in default skin 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5234 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-10-01 13:07:24 +00:00
lotus
93ddf206e6 opensearch fix if user agent had no language 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5233 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-09-30 20:13:18 +00:00
lotus
b8538fae04 search form like on result page 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5232 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-09-30 19:59:55 +00:00
lotus
3cce13d1b7 more compact search form 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5231 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-09-30 18:03:52 +00:00
lotus
95fddf056c - better support for narrow windows on searchpage 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5230 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-09-30 17:50:42 +00:00
orbiter
c8bdd965ec - larger update time for status page 
- balancer writes cause of robots.txt in log file for crawl delay
- removed log output for forced GC
- smaller RAM flush for RWI cache, should cause more usage of cache and faster crawling

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5228 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-09-30 11:09:46 +00:00
lotus
3a919bf24e better solution for search result layout 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5227 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-09-29 19:11:16 +00:00
lotus
f95ec8b813 fix for non-accessible 2nd-line tabs on admin console 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5226 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-09-29 18:33:38 +00:00
orbiter
c44e97d6dd more lines in log on status page 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5225 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-09-29 18:07:55 +00:00
orbiter
dc149df3b1 new status page layout: 
- smaller kaskelix image to make room for more information
- added the memory graph, since this picture is widely used to monitor YaCys activities
- added border to log line iframe (looks better together with memory graph)

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5224 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-09-29 14:59:06 +00:00
daburna
298196e7a4 translation update 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5223 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-09-29 13:24:03 +00:00
lotus
dda771db9d - search result layout 
- tray only for windows

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5222 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-09-29 12:39:57 +00:00
apfelmaennchen
7f88dadf87 YaCy-UI: added a notice to a bug in bookmarks folder recursion! 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5221 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-09-29 06:02:48 +00:00
orbiter
2126315d40 layout changes for search result page 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5220 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-09-28 23:19:56 +00:00
low012
f7003aa75f *) updated whos.amung.us code in terminal page 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5217 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-09-28 12:54:01 +00:00
apfelmaennchen
cc9f1e9dd9 YaCy-UI: enabled edit bookmarks in toolbar 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5215 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-09-27 21:15:30 +00:00
apfelmaennchen
00c843f7d9 YaCy-UI: 
- added scrollbar to folder widget
- slightly improved Ajax handling for folder and tag widget

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5214 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-09-27 11:38:20 +00:00
lotus
9d4f0325e1 - removed shutdown from search page (we have it in tray now!) 
- fixed doubleclick action for tray

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5211 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-09-26 10:55:08 +00:00
apfelmaennchen
f293554216 XBEL: small bug fix 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5208 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-09-25 05:26:36 +00:00
apfelmaennchen
848dbbd280 YaCy-UI: 
- added bookmark folder widget to sidebar (it uses the xbel.xml with XSLT!

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5206 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-09-24 21:09:12 +00:00
apfelmaennchen
7462b5db7b /xml/bookmarks/xbel/xbel.xml: 
- added support for xsl-style sheets: style=/path-to-xsl/style.xsl
- removed fixed root folder "YaCy Bookmarks"

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5205 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-09-24 21:04:27 +00:00
orbiter
6e7d113eac fix for wrong index initialization after network switch 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5203 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-09-23 23:30:25 +00:00
orbiter
7b35d54c6c fixed some problems with network switching (was not completely 'clean') 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5200 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-09-23 12:11:19 +00:00
apfelmaennchen
8b6a0376db YaCy-UI: 
- another XSS fix (TagCloud)
- upload of missing icons for toolbar

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5198 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-09-22 05:19:37 +00:00
apfelmaennchen
2ff6f0360e YaCy-UI: small fixes to accomodate Safari & Opera 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5196 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-09-21 14:26:35 +00:00
apfelmaennchen
a28af9d4c4 YaCy-UI: enabled language selection - special thanks to orbiter 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5195 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-09-21 10:49:05 +00:00
apfelmaennchen
1d9f0234c8 YaCy-UI: 
- added basic functionallity to add bookmark(s) from search results
- re-design of sidebar search form
- replaced classic search with compare search 

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5194 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-09-21 10:04:51 +00:00
orbiter
1198eeecc7 added language selection to search query: 
- the language can be selected using a LANGUAGE:<language> element in the query line, i.e.:
java LANGUAGE:en
- the language can be selected with a post element in google-style syntax with the 'rl' element:
?lr=lang_en&query=java

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5193 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-09-21 07:28:57 +00:00
orbiter
00c1535f84 added ranking and evaluation of language type in a search 
the wanted language is taken from the browser user-agent string

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5192 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-09-21 00:04:42 +00:00
apfelmaennchen
e201ad0e9f YaCy-UI: 
- moved search result to flexigrid
- added a toolbar similar to YaCyBar

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5191 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-09-20 22:21:56 +00:00
apfelmaennchen
06c07f2e0f YaCy-UI: fixed broken style for add bookmark dialog 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5190 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-09-20 19:33:20 +00:00
apfelmaennchen
a5f61e8b1b YaCy-UI: 
- fixed xss vulnurability
- improved add bookmark dialog

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5189 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-09-20 18:28:58 +00:00
lotus
a81cb78211 finally some putHTML on htroot/xml/ 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5188 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-09-20 07:55:30 +00:00
orbiter
bfcf9b7aa3 - added language detection using metadata from documents: html and odt documents provide this information 
- metadata and results from statistical analysis are compared and result is printed out as debug lines
- added ranking profile for wanted language
- added class with ISO 639 table, a list of all valid country codes that will be used for the language identification

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5187 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-09-19 22:19:11 +00:00
apfelmaennchen
3768a1bd32 set encoding="UTF-8" for getpageinfo_p.xml 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5186 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-09-19 14:29:10 +00:00
apfelmaennchen
5e8bd0f29c small fixes to getpageinfo_p.xml and htmlFilterContentScraper.java with respect to keyword extraction 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5185 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-09-19 14:27:44 +00:00
lotus
029e16b653 replaced some put(String, String) by putHTML(String, String) on serverObjects respond 
in htroot/ root
didn't touch htroot/xml/
this should solve potential xss issues

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5184 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-09-19 11:45:11 +00:00
apfelmaennchen
5b2a57bfd0 - /xml/util/getpageinfo_p.xml added <desc> and <lang> tags 
- changed htmlFilterContentScraper.getKeywords() to split either space or comma charater not both

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5183 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-09-18 21:01:23 +00:00
orbiter
ea5de7436d added Sciencenet to the compare search 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5178 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-09-18 10:56:18 +00:00
orbiter
1bbf362cef update to the crawl balancer: better organization and better crawl delay prediction 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5176 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-09-17 21:45:21 +00:00
orbiter
ddcf285499 - fixed a bug in performance setting (did not work with german translation) 
- reduced maximum number of error url references to save some memory (this was actually a small memory leak)

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5174 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-09-16 23:04:24 +00:00
orbiter
0cd0fee546 fixed bug with wrong proxy result enqueueing. See: 
http://forum.yacy-websuche.de/viewtopic.php?p=8130#p8130
- removed the online status property. This influenced the proxy behavior and created some complexity that was not needed because the online status was never used as it was ceated for (offline browsing)
- checked all proxy identification procedures during crawling and enhanced transparency and error checking
- fixed a proxy identification routine that caused the wrong selection of the proxy result queue

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5173 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-09-16 21:56:23 +00:00
apfelmaennchen
e071f759d2 YaCy-UI: small optical changes 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5172 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-09-16 21:39:14 +00:00
apfelmaennchen
bbacf86fe8 - added /xml/bookmarks/posts/add_p.xml 
- security fix to /xml/bookmarks/posts/delete_p.xml
- YaCy-UI: added 'add' and 'delete' bookmarks

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5171 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-09-16 21:38:13 +00:00
apfelmaennchen
cd1ac5bb90 - fixed security issue with /xml/util/ynetSearch.xml 
- hopefully fixed YaCy-UI local search with async=false for Ajax-request

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5170 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-09-16 05:55:31 +00:00
orbiter
c73cf05ddd tried to fix local search in yacy-ui 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5169 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-09-15 21:56:53 +00:00
apfelmaennchen
99ff478d63 YaCy-UI: small optical fix 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5167 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-09-15 20:30:33 +00:00
orbiter
7e24c51fd5 - removed alternative search page in main menue in favor of rich client search page 
- added necessary option to search request of yacy-ui to get snippets

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5165 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-09-15 19:00:22 +00:00
apfelmaennchen
7c5867a832 Major update to YaCy-UI 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5162 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-09-15 17:18:07 +00:00
lotus
73f233bb11 * set resource observer to 1000MB 
* transparent favicon

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5160 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-09-15 12:41:27 +00:00
orbiter
1be24158a2 small fix / rendering option 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5159 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-09-15 10:22:05 +00:00
orbiter
105be67998 - some bugfixes to compare search 
- redesigned input boxes: smaller, more space for result page

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5158 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-09-15 09:43:51 +00:00
orbiter
693fa2a157 - renamed Comparison to compare_yacy 
- added more search engines
- some refactoring and added a list that is used to present the search engine list in a specific order
- added simpleheader and no-header options
- added the compare search to the simple header
- added default compare search page selection storage - after re-start you get the same default search engines as you selected before

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5157 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-09-15 09:17:05 +00:00
lotus
6216105ca5 small fix 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5150 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-09-14 18:12:26 +00:00
lotus
5e5178b5e8 please use putHTML to avoid XSS 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5149 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-09-14 18:08:39 +00:00
low012
b33a6cbb77 *) less disturbing elements in yacy frame 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5148 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-09-14 17:58:08 +00:00
f1ori
98d902b972 * remember last searchwords in Comparison_p.html 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5147 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-09-14 15:21:57 +00:00
orbiter
bd45c5a2bc integrated the comparison page into the main menu 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5146 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-09-14 10:40:01 +00:00
low012
5e0390a24c *) Ooooooooops! 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5145 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-09-13 17:14:41 +00:00
low012
dc56c35289 *) added page to compare results of 2 search engines 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5144 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-09-13 16:50:01 +00:00
lotus
880d1a83e2 do not change memory and some non-defined tasks with performance profiles 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5142 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-09-12 11:54:25 +00:00
lotus
b68d06a6e8 performance settings based on network's remote crawl speed 
removed some _pro values from config

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5134 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-09-10 12:52:17 +00:00
orbiter
bb5c898441 enhancements to localsearch behavior 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5131 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-09-09 10:24:42 +00:00
orbiter
3c6e8d2015 set default ppm when network is switched 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5127 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-09-08 18:20:05 +00:00
orbiter
20c2d3c248 fix for bad formatting in CrawlResults 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5126 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-09-08 13:59:35 +00:00
lotus
b92105c8b0 do not change auto recrawl scheduler with performance profiles 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5123 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-09-07 13:59:24 +00:00
orbiter
d09ddabd09 corrected a design mistake (5-byte hashes not necessary) 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5119 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-09-04 21:28:00 +00:00
orbiter
c97d0fcee7 modified the domain list export function: 
- used the new superfast domain list generation from the domain statistics
- better interactive behavior

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5118 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-09-04 20:28:36 +00:00
orbiter
77ee0765a4 - added domain statistic generation to IndexControlURLs_p.html servlet 
- added 'delete all' button to all results of such a domain statistic output which causes that all urls to this domain are deleted
- extended stack cleaner to clean also the statistics: they are not completely destroyed, only the smallest counting domains are removed


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5117 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-09-04 19:41:57 +00:00
lotus
44bc8311af translation fix 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5116 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-09-04 19:26:59 +00:00
lotus
e5c0b969d6 * save performance profile speed 
* fix for wrong javastart_priority after first start

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5115 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-09-04 19:12:22 +00:00
lotus
d7a16c1f30 * added shutdown on search page (this page is shown after clicking the tray icon) 
* shorter, less technical words for configuration-links

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5114 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-09-04 12:51:05 +00:00
orbiter
80a7bc93d6 - added statistical evaluation about domains that appear during crawling 
- added tables that show this statistics in CrawlResults web pages

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5113 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-09-04 09:59:17 +00:00
orbiter
4a4f388ca5 re-design and simplification of crawl start menu layout 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5112 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-09-04 07:56:29 +00:00
orbiter
4fbee21cea - added fetch-ahead again (had been removed in last commit) 
- reverted default query mode to verify=false

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5111 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-09-03 23:50:13 +00:00
orbiter
fc03b0437a fixed a error case where a second search after a first search with a different search word failed 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5109 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-09-03 15:55:25 +00:00
daburna
992635c074 translation update 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5107 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-09-03 13:44:58 +00:00
orbiter
05dbba4bab added logging conditions to all fine and finest log line calls 
this will prevent an overhead for the generation of the log lines in case that they then are not printed

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5102 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-09-03 00:30:21 +00:00
orbiter
d3d41e2ee4 - fixed problem with searching with quotes (still not complete, but not as bad as before) 
- fixed parsing of crawl-delay statements when seconds were given with float numbers
- enhanced performance of profiling (not too many loggings; not more than one per second)
- removed some debug output
- fixed wrong return type in logging
- added a logging condition in httpd to prevent that logging statements are generated when they are not written (should be added everywhere!)
- fixed wrong word distance computation in RWI management


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5101 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-09-02 23:49:48 +00:00
lotus
3fbfd5a78b * fix for non-changing offset on new search term 
* dht-heap doesn't has to be deleted (5097), we simply write a new one on exit
* do not install YaCy in startup because a Windows-shutdown might corrupt something. Installing YaCy as a service would solve this.

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5099 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-09-02 15:09:31 +00:00
danielr
cd19d0aee6 - added warnings for failed transferRWI (dht-in) 
- fixed parseMultipart (uncompress gzipped body) (dht-in)
- fixed parseMultipart (using content-length only if uncompressed)
- better gzipped POST (chunked instead of content-length) (dht-out)



git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5096 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-08-29 09:42:39 +00:00
orbiter
016f57d714 fixed a dead link 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5094 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-08-28 21:45:58 +00:00
f1ori
b8ee04daf1 fix for http://www.yacy-forum.org/viewtopic.php?f=2&t=160 (wrong url in form) 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5086 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-08-26 18:45:19 +00:00
lotus
d9d9c522a1 addendum to last commit 
moved recrawl times for standard profiles to constants
calculate new specific dates in cleanup job

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5082 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-08-26 13:20:18 +00:00
lotus
480497f7c9 changed recrawl 
use a specific date to define old documents
this solves an unwanted recrawl-loop during a running crawl

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5081 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-08-25 20:31:32 +00:00
orbiter
536e77e8b7 modifications towards a single database operation to read/write http header and cached file at once: 
- removed distinction between header file types for http and ftp; ftp is simulated by using http properties
- removed all old resourceInfo classes that handled this distinction
- introduced a new distinction between http request and http response objects
- unified new response objects with two other object types that had been introduced elsewhere
- changed all servlet call methods to use the new http request header object type
- divided static object keys for http header properties into request and response types
- refactoring here and there (a large number of type changes and many methods merged/moved)


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5079 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-08-25 18:11:47 +00:00
apfelmaennchen
296fa2265b YaCy-UI: removed unused Servlet ymarks.java 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5076 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-08-24 08:47:37 +00:00
apfelmaennchen
a9cf3b42c4 YaCy-UI: removed unused JavaScript 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5075 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-08-24 08:43:07 +00:00
apfelmaennchen
08bf3fd235 YaCy-UI: updated to jQuery-ui 1.6b 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5074 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-08-24 08:41:52 +00:00
apfelmaennchen
58d7e6f1a6 - some small, rather optical changes to bookmarks 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5071 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-08-22 18:19:21 +00:00
apfelmaennchen
8551e1d106 - added Info link (/xml/util/getpageinfo_p.xml) to bookmarks 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5069 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-08-21 21:28:57 +00:00
apfelmaennchen
8d1bedfc3a - added bookmarkTitle to CrawlStart_p.html 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5068 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-08-21 21:07:21 +00:00
danielr
3c68905540 remove redundant null checks 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5065 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-08-20 08:37:39 +00:00
danielr
753a1ae430 - changed default browser from netscape to firefox 
- fixed "Inefficient use of keySet iterator instead of entrySet iterator" [WMI_WRONG_MAP_ITERATOR, FindBugs]
- fixed some possible null pointer accesses


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5063 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-08-20 07:54:56 +00:00
orbiter
7989335ed6 Preparations to replace the HTCache with a new storage data structure: 
- refactoring of the HTCache (separation of cache entry)
- added new storage class for BLOBs. (not used yet, this is half-way to a new structure)

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5062 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-08-19 14:10:40 +00:00
f1ori
c05edba6b8 * added text/html-URL to opensearchdescription, so one-click-installation in firefox is possible 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5060 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-08-17 21:13:41 +00:00
danielr
8422ee5ec4 - fixed UnsupportedEncoding (in proxy) using defaultCharset if no characterEncoding can be determined 
- serverFileUtils.copy* use now Charset instead of String
- added some warnings for ignored exceptions


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5043 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-08-09 12:00:31 +00:00
danielr
621b473b18 * removed some warnings of findbugs (http://findbugs.sf.net) 
- removed unnecessary code (unused variables, String.toString)
- corrected some calculations (cast int to double or long ;)
- improved little performance (using Integer.valueOf() instead of new Integer)
- log if some File-actions fail (mkdir(), delete(), ...) and some ignored exceptions
- finalized some (more) fields
- finally close some streams
- made inner classes static if not using environment
- generalized some equals (from specificClass to Object)
- fixed some potential nullpointer accesses


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5039 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-08-06 19:43:12 +00:00
apfelmaennchen
e1574fe02e - added autoReCrawl folders to bookmarks (DATA/SETTINGS/autoReCrawl.conf) 
- the serverBusyThread checks folders every 60 min. (==> autoReCrawl_idlesleep in yacy.conf)
- added option to create bookmarks from CrawlStart URL

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5033 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-08-04 20:43:36 +00:00
orbiter
ebb40d324b enhanced memory chart: shows now also the size of the word cache as third vector. 
The PPM is now shown without a scale, but with a new anotation at the chart entry.

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5032 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-08-04 10:47:26 +00:00
danielr
17b7845eb5 * refactoring 
- moved constants from plasmaSwitchboard to own class (all 232 ;)
- moved remoteProxy-Methods to httpRemoteProxyConfig, better names
- removed some unnecessary code (else-statements)
* formatting (correct indentation)
* minor bugfixes (due to findbugs.sf.net)
* hopefully fixed "missing quote" (announcing StringParts as UTF-8)


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5031 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-08-02 13:57:00 +00:00
danielr
3bb870bfcd added final where possible 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5030 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-08-02 12:12:04 +00:00
f1ori
b0724e5ec0 * add config option to disable cookie monitoring (disabled by default) 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5028 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-07-30 21:19:06 +00:00
lotus
f0dfb858d8 reverted balloon messages from r5014. don't try to annoy the user. ;) 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5027 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-07-30 07:42:02 +00:00
lotus
cafa1a952f - performance settings refactoring: made headlines to fieldsets 
- show performance profiles more prominent for easier configuration

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5026 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-07-30 07:37:11 +00:00
f1ori
6952b13c08 * fix automatic decrementing accepted remote crawl limit: 
-> http://forum.yacy-websuche.de/viewtopic.php?f=5&t=1355


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5025 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-07-29 18:52:48 +00:00
f1ori
d02a26d642 * added lock icon to ConfigAppearance menu item 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5024 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-07-28 21:09:19 +00:00
f1ori
0baa8d651e * marked some text as labels for checkboxes 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5022 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-07-27 11:39:42 +00:00
lotus
f9fb2d0fee hide memory tables by default 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5020 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-07-27 06:59:24 +00:00
lotus
d42eae25f8 yacyTray: 
fix for unproper shutdown
some messages

installer:
start shortcuts minimized

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5014 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-07-24 06:49:30 +00:00
orbiter
5f41913043 fix for http://forum.yacy-websuche.de/viewtopic.php?f=6&t=1347&hilit=&p=9216#p9216 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5012 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-07-20 21:52:51 +00:00
orbiter
c3d461d191 - removed superfluous copyright statement 
- updated my email address

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5011 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-07-20 17:14:51 +00:00
orbiter
3ca98fee42 removed superfluous copyright statement 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5010 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-07-20 00:21:07 +00:00
danielr
af616a92ff fixed welcome message above image on custom search page 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5007 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-07-18 13:41:06 +00:00
danielr
10d01fdd0d addendum to 5002 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5005 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-07-16 09:47:48 +00:00
danielr
d14e8d348f - times of LOCAL_SEARCH are shown in milliseconds (also in yacysearch.java ;) 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5003 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-07-15 17:35:02 +00:00
danielr
b63a519456 fixed build problem 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5001 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-07-14 17:44:49 +00:00
danielr
e12438142e - warning instead of NPE if urlHash is not in index of CrawResults 
- times of LOCAL_SEARCH are shown in milliseconds


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5000 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-07-14 09:12:58 +00:00
lulabad
5784070f6e added user agent 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4999 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-07-14 08:38:11 +00:00
lotus
b704cf1fbc * no need to update the searchpage links anymore 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4994 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-07-13 07:55:20 +00:00
lotus
f8a1e3175e new yacyTray 
this will make a YaCy icon in the tray area on supported platforms
enabled by default
the search page will open on double click

used JDIC 0.9.4 from https://jdic.dev.java.net/

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4992 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-07-13 07:51:45 +00:00
daburna
4a7e5f377e added translation for new customization pages 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4991 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-07-12 12:27:33 +00:00
orbiter
f7aaeb3fad created new main menu entry 'Customization and Integration' 
- moved some already existing servlets to this menu
- renamed the skin servlet to appearance
- added a set-to-default-button to the search page appearance setting
- removed the peer profile servlet which is now replaced by a field in the new appearance servlet

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4980 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-07-10 19:57:09 +00:00
orbiter
1e6d12f146 Major update to BLOB data structures: 
- introduced a new BLOB file format: kelondroBLOBHeap. This is a flat file with an index in RAM.
  very similar to the eco-tables, but with flexible value sizes. It will replace the kelondroBLOBTree,
  which is based on a kelondroTree, a file-AVL-based index data structure.
- the HTCACHE header file was replaced by the new blob heap file structure
- the robots.txt file was replaced by the new blob heap file structure
- the robots parser was enhanced (bugfixing for double-loading of the same robots.txt)
- other BLOB-dependent data structures were prepared to use also the new BLOB heap
- fixed a bug in the snippet fetch process: the file header was not written to the header index
There should now be less IO during snippet fetch and during crawling


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4978 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-07-10 00:47:37 +00:00
orbiter
b38f467e3c better SRU compliance 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4976 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-07-07 21:50:24 +00:00
f1ori
3980dd7554 * maske ampersan & Co. in xbel export 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4975 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-07-07 21:30:11 +00:00
orbiter
a6719dfd2b - refactoring of robots parser 
- no more keep-order parameter in remove (it was not possible to make this strict, and not useful)
- some small enhancements in balancer
- robots parser without references in switchboard
- changes synchronization in robots

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4969 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-07-05 00:35:20 +00:00
orbiter
e81be7d4f2 added many missing user-agent declarations for yacy http client connections. 
the most important fix was the addition of the yacybot user-agent for robots.txt loading,
because web masters look for that access to see if the crawler behaves correctly.

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4968 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-07-04 11:03:03 +00:00
orbiter
474e29ce4a added options to configure the 'corporate identity'-icons, the home page link and the greeting line from 
the skin menue. Additionally an example is given there how to integrate a search page with an iframe.
Please see the skin menu.

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4967 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-07-03 23:37:04 +00:00
orbiter
474659a71f - modified and enhanced the crawl balancer: better list export, fixing of damaged crawl queue at start-up, re-sorting at start-up to enhance domain order 
- added option to set minimum crawl delta for domains in balancer
- added default values to crawl deltas in yacy.init
- added configuration for these deltas in performance queues
- enhanced performance setting computation (more time for indexing queue for a faster flush
- remote crawling is now enabled during local crawling if indexer has space and time for more links
- added database stub for new distributed file system
- refactoring of time computation to get an abstraction level that will be used by a TTL rule in new distributed file system

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4966 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-07-03 13:08:37 +00:00
lotus
1afc36d9a8 * update to JRE 6u6 
* fix for proper translation

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4965 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-07-02 13:36:42 +00:00
orbiter
8e179f6588 removed option to do a re-crawl with a period of minutes. Such a short time does not make sense and it may cause endless indexing loops. The removing of the option will ensure that a misuse is prevented. 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4964 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-07-01 23:47:33 +00:00
danielr
da917cf4b1 undo reduced menu 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4960 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-06-30 07:11:13 +00:00
danielr
0c1dc703e4 - set staticIP at startUp 
- added setting for reduced menu (simpleMenu)


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4959 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-06-29 18:35:15 +00:00
lotus
f6b92456c0 confirm bookmark delete 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4958 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-06-27 05:48:29 +00:00
lotus
4a53649ee7 fixed dht-urls and ranking distribution log statistics 
* NOTE: please have in mind that there can be whitespaces in pathnames

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4956 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-06-26 07:12:03 +00:00
lotus
8d83185cb4 fixed dht-chunks/protocol log statistics 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4955 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-06-25 08:15:07 +00:00
danielr
63eadfdf84 fixed unlimited FileSizeLimit 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4954 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-06-24 19:11:27 +00:00
lotus
2dc7c00c1c fixed indexing log statistics 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4953 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-06-24 07:01:04 +00:00
low012
0a956d1b11 *) removes focus on text field on search result page (http://forum.yacy-websuche.de/viewtopic.php?f=9&t=1257) 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4945 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-06-20 16:46:30 +00:00
danielr
01e9c7320e PerformanceProfile: don't change peerPing, peerSeed and cleanUp 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4938 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-06-19 12:39:45 +00:00
orbiter
c998dc6556 - added security functions to flush url and search caches in case that memory is full 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4933 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-06-16 21:39:58 +00:00
lotus
877299cc74 better installer on Windows Vista 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4927 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-06-14 18:34:12 +00:00
danielr
68c38c2d34 - WatchCrawler shows status without JavaScript 
- Performance can be scaled + DHT-profile
- names for pool-threads
- some small refactorings


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4923 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-06-14 10:24:58 +00:00
lotus
fc79f013c4 better solution to update shortcut 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4920 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-06-12 20:04:32 +00:00
orbiter
994c609cf8 added new shell script to do a web search from the terminal 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4916 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-06-11 21:33:36 +00:00
lotus
48edbef5c7 * fix: display proper port on 1st startup 
* new message on portchange
* first implementation of external link-update for search page (still inactive)

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4915 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-06-11 19:04:39 +00:00
orbiter
f5ef7f222e - fixed a bug in parser (directory paths had not been recognized) 
- no access check when a search is made only local without snippet fetch
- added comment and status message in resourceObserver (this takes very long at startup time!)

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4911 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-06-11 09:54:58 +00:00
danielr
48f87db7f1 added performance profiles (to be fine-tuned) 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4910 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-06-10 19:23:38 +00:00
daburna
73051ecb37 corrected wrong link 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4908 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-06-09 14:30:16 +00:00
det
af8f7b8bba add image rising while mouse over in Collage 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4898 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-06-07 18:33:04 +00:00
lotus
260553c3a5 better messages 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4897 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-06-07 14:24:11 +00:00
danielr
7feae906aa - organize imports 
- removed potential null pointer accesses
- removed unnecessary casts


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4893 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-06-06 16:01:27 +00:00
det
f597185026 Initial import of the resource observer framework 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4892 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-06-06 13:10:21 +00:00
orbiter
c10eaf9bdb - fix for pop-up page upon first start 
- added comments in opensearchdescription to explain fast mode
- release 0.59

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4890 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-06-05 20:46:31 +00:00
orbiter
0e1c7dfaaf small fix for testing environment that should not affect a production environment 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4886 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-06-05 13:08:40 +00:00