Commit Graph

2411 Commits

Author SHA1 Message Date
orbiter
28d2d28573 added support for filetype search
(just use filetype:<type> in the search query)

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5418 6c8d7289-2bf4-0310-a012-ef5d649a1542
2008-12-29 17:57:04 +00:00
orbiter
78c568331e added test channel to /xml/feed.rss
can be obtained with 
http://localhost:8080/xml/feed.rss?set=TEST
returns always a single feed entry with a fresh date

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5416 6c8d7289-2bf4-0310-a012-ef5d649a1542
2008-12-29 12:39:07 +00:00
orbiter
e004da48d3 - added fast fingerprint computation for files (any). Will be used in new index dump method
- refactoring

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5415 6c8d7289-2bf4-0310-a012-ef5d649a1542
2008-12-29 12:22:13 +00:00
low012
eab72424df *) Fixed small bug: When adding new elements to blacklist via import, the blacklist which the elements were added to was supposed to be displayed, which did not work correctly.
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5414 6c8d7289-2bf4-0310-a012-ef5d649a1542
2008-12-28 09:58:02 +00:00
low012
0e56675596 *) cleaning up ;-)
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5413 6c8d7289-2bf4-0310-a012-ef5d649a1542
2008-12-27 20:09:36 +00:00
low012
cf69557ea2 *) blacklists can be exported as XML or plain text now
*) blacklist import via file upload works now

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5412 6c8d7289-2bf4-0310-a012-ef5d649a1542
2008-12-27 15:38:20 +00:00
low012
1594a15be9 *) explicit mentioning of blacklist in blacklist cleaner
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5411 6c8d7289-2bf4-0310-a012-ef5d649a1542
2008-12-27 13:06:05 +00:00
low012
5a89266598 *) new parameters for future use (better blacklist handling for im- and export)
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5403 6c8d7289-2bf4-0310-a012-ef5d649a1542
2008-12-19 19:33:08 +00:00
orbiter
e34ac22fbd - added new monitoring servlet at
http://localhost:8080/PerformanceConcurrency_p.html
- used the new monitoring to do some fine-tuning of the indexing queue

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5402 6c8d7289-2bf4-0310-a012-ef5d649a1542
2008-12-19 15:26:01 +00:00
orbiter
d376d81fc4 replaced busy thread control of crawl stacker by blocking threads
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5400 6c8d7289-2bf4-0310-a012-ef5d649a1542
2008-12-18 23:18:34 +00:00
orbiter
7535fd7447 - refactoring of CrawlEntry and CrawlStacker
- introduced blocking queues in CrawlStacker to make it ready for concurrency
- added a second busy thread for the CrawlStacker
The CrawlStacker is multithreaded. It shall be transformed into a BlockingThread in another step.
The concurrency of the stacker will hopefully solve some problems with cases where DNS blocks.

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5395 6c8d7289-2bf4-0310-a012-ef5d649a1542
2008-12-17 22:53:06 +00:00
lotus
6569cbbec1 npe fix: http://forum.yacy-websuche.de/viewtopic.php?t=1646
(break to avoid bad side effects)

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5394 6c8d7289-2bf4-0310-a012-ef5d649a1542
2008-12-16 20:53:31 +00:00
orbiter
2802138787 - refactoring of CrawlStacker (to prepare it for new multi-Threading to remove DNS lookup bottleneck)
- fix of shallBeOwnWord target computation heuristic


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5392 6c8d7289-2bf4-0310-a012-ef5d649a1542
2008-12-15 00:02:58 +00:00
lotus
b1e211b258 no error-alert: http://forum.yacy-websuche.de/viewtopic.php?t=1639
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5391 6c8d7289-2bf4-0310-a012-ef5d649a1542
2008-12-13 12:04:08 +00:00
orbiter
13cb0916ee changes to statistics and content of thread dump servlet
(points now more directly to performance leaks without mentioning class calls inside of sun/java calls that cannot be changed anyway)

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5390 6c8d7289-2bf4-0310-a012-ef5d649a1542
2008-12-11 20:13:14 +00:00
orbiter
e1acdb952c fix for problem with userDB and bookmarksDB which was caused by changes in kelondroRA in SVN 5376
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5385 6c8d7289-2bf4-0310-a012-ef5d649a1542
2008-12-08 00:17:45 +00:00
lotus
e918d64c23 show hand-cursor an labels
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5383 6c8d7289-2bf4-0310-a012-ef5d649a1542
2008-12-06 17:32:53 +00:00
orbiter
4a2dac659e more speed hacks:
- modified and activated write buffer
- increased cache flush factor
- fixed a problem with deadlocking of indexing process

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5382 6c8d7289-2bf4-0310-a012-ef5d649a1542
2008-12-05 13:55:48 +00:00
lotus
1fb518a5b4 display <String> etc.
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5380 6c8d7289-2bf4-0310-a012-ef5d649a1542
2008-12-04 20:21:53 +00:00
orbiter
47292e696a more performance hacks
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5379 6c8d7289-2bf4-0310-a012-ef5d649a1542
2008-12-04 12:54:16 +00:00
orbiter
bd1dc9cd5d thread dump with statistics, a little bit of profiling
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5377 6c8d7289-2bf4-0310-a012-ef5d649a1542
2008-12-03 23:26:25 +00:00
orbiter
d39d420b39 performance hacks
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5376 6c8d7289-2bf4-0310-a012-ef5d649a1542
2008-12-03 15:38:29 +00:00
lotus
5280ad638d added basic performance page
other performance settings can be found on advanced settings

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5375 6c8d7289-2bf4-0310-a012-ef5d649a1542
2008-12-03 14:10:01 +00:00
lotus
1a51d9fcfd display proper values
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5374 6c8d7289-2bf4-0310-a012-ef5d649a1542
2008-12-02 17:57:30 +00:00
orbiter
0b4808ba3d added new interactive search feature:
- during the user types search queries, the local database is searched
- results are presented interactively

This was implemented using a new JSON result format for search results in YaCy
- added JSON as file format for servlets
- refactoring of current search servlets (xml and html)
- added JSON output format for search results
- added AJAX-based search page, that uses the yacysearch.json selrvlet to print results as a query is typed

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5373 6c8d7289-2bf4-0310-a012-ef5d649a1542
2008-12-02 15:24:25 +00:00
lotus
fea82b54ef more contrast on search snippets
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5370 6c8d7289-2bf4-0310-a012-ef5d649a1542
2008-11-26 19:57:13 +00:00
lotus
1951d30a62 addendum to last commit
handle words with length < 3 correctly

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5369 6c8d7289-2bf4-0310-a012-ef5d649a1542
2008-11-26 19:43:40 +00:00
lotus
325ba7bfb8 only query words with length > 2
this is not complete, yet

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5368 6c8d7289-2bf4-0310-a012-ef5d649a1542
2008-11-26 16:41:38 +00:00
lotus
489edb4473 improved pattern selection
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5367 6c8d7289-2bf4-0310-a012-ef5d649a1542
2008-11-26 10:06:38 +00:00
low012
e423fa9846 *) added method to only get file names in directory listing which match a filter
*) only files which end with .black will be listed as blacklists
*) added a little bit of Javadoc

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5366 6c8d7289-2bf4-0310-a012-ef5d649a1542
2008-11-25 20:26:06 +00:00
lotus
577b53aee6 added more search engines
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5365 6c8d7289-2bf4-0310-a012-ef5d649a1542
2008-11-24 13:05:20 +00:00
lotus
7f4d411c0d npe-fix
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5364 6c8d7289-2bf4-0310-a012-ef5d649a1542
2008-11-24 13:04:57 +00:00
lotus
1545e5440a * index deletion: checkbox-confirmation
* watch crawler: less load on exhausted peers; wait for data before reloading again

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5359 6c8d7289-2bf4-0310-a012-ef5d649a1542
2008-11-23 12:02:58 +00:00
orbiter
10f5ec1040 reverted last commit (more testing needed)
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5356 6c8d7289-2bf4-0310-a012-ef5d649a1542
2008-11-22 00:12:50 +00:00
daburna
ba5b274b8c #translation update:
-blacklist
-crawlstart
...

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5353 6c8d7289-2bf4-0310-a012-ef5d649a1542
2008-11-21 16:45:45 +00:00
orbiter
0ca4bc7b79 - added reader and visualization for mediawiki-export files:
files exported from mediawiki using the xml schema according to
http://www.mediawiki.org/xml/export-0.3/
can be processed to be viewed in a YaCy servlet.
To acces such a file, place it into
DATA/HTCACHE/mediawiki/
i.e. the export from german wikipedia would be:
DATA/HTCACHE/mediawiki/wikipedia.de.xml
This file can then be accessed using the URL
http://localhost:8080/mediawiki_p.html?dump=wikipedia.de.xml&title=YaCy
if this is done the first time, an index file is created
(for this case: more than 4 million lines must be written, this takes about 15 minutes)
Then try the same url again.

- enhanced also the md5 computation speed


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5352 6c8d7289-2bf4-0310-a012-ef5d649a1542
2008-11-20 18:31:52 +00:00
lotus
4f996a7651 fix for logparser pattern
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5349 6c8d7289-2bf4-0310-a012-ef5d649a1542
2008-11-17 16:23:17 +00:00
orbiter
867d0f2f56 removed some unnecessary pause delays
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5346 6c8d7289-2bf4-0310-a012-ef5d649a1542
2008-11-14 23:36:33 +00:00
lotus
fd83e59f8e new remote search average
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5343 6c8d7289-2bf4-0310-a012-ef5d649a1542
2008-11-14 11:50:46 +00:00
orbiter
dba7ef5144 extended crawling constraints:
- removed never-used secondary crawl depth
- added a must-not-match filter that can be used to exclude urls from a crawl
- added stub for crawl tags which will be used to identify search results that had been produced from specific crawls
please update the yacybar: replace property name 'crawlFilter' with 'mustmatch'.
Additionally, a new parameter named 'mustnotmatch' can be used, which should be by default the empty sring (match-never)

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5342 6c8d7289-2bf4-0310-a012-ef5d649a1542
2008-11-14 09:58:56 +00:00
orbiter
0ae84f4f8e set some default values for a crawl start that should cause less confusion and mistakes
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5334 6c8d7289-2bf4-0310-a012-ef5d649a1542
2008-11-12 19:48:22 +00:00
lotus
4745e89451 auto-choose crawl type
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5331 6c8d7289-2bf4-0310-a012-ef5d649a1542
2008-11-12 14:44:23 +00:00
low012
421d056550 *) changed layout of blacklist adminstration (less cluttered)
*) it is possible to move/edit/delete more than one entry at a time now
*) it is easier to choose a target for blacklist import now
*) fixed several bugs
*) to be continued...

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5330 6c8d7289-2bf4-0310-a012-ef5d649a1542
2008-11-12 00:47:54 +00:00
orbiter
674ad2d55b different handling of error cases that occur during loading files with http or ftp:
methods throw exception instead of returning an error string

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5328 6c8d7289-2bf4-0310-a012-ef5d649a1542
2008-11-11 21:33:40 +00:00
f1ori
ae80f3e6a5 * extend opensearchdescription to support compare_yacy.html
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5324 6c8d7289-2bf4-0310-a012-ef5d649a1542
2008-11-09 00:23:19 +00:00
orbiter
1b18d4bcf3 enhancement to crawling and remote crawling:
- for redirector and  remote crawling place crawling url on notice queue instead of direct enqueueing in crawler queue
- when a request to a remote crawl provider fails, remove the peer from the network to prevent that the url fetcher gets stuck another time again

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5320 6c8d7289-2bf4-0310-a012-ef5d649a1542
2008-11-06 12:30:55 +00:00
orbiter
3f746be5d4 - consolidation and refactoring of many DHT target - computing methods
- implemented vertical DHT acceptance ("my own DHT") to accept new targets
- added new target computation for global search: addresses vertical targets also
- enhanced remote crawling: collection of remote crawl urls if queue has less than 100 entries (was: 0 entries)
- better performance value computations for PPM selection in network configuration

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5319 6c8d7289-2bf4-0310-a012-ef5d649a1542
2008-11-06 10:07:53 +00:00
orbiter
d014b2728a Design-check, Extension and Refactoring of DHT target position computation:
- two different computations (but mathematical equivalent) of the DHT distance had been consolidated
- moved from 0.0 .. 1.0 double-range position computation to 0 .. Long.Max range for DHT targets
- added fast Long - to - hash computation
- high-precision target computation of gaps for new peers
- added new target computation for horizontal and vertical DHT targets (not yet in use)
- old horizontal-only DHT targets will be upwards compatible to new horizontal and vertical DHT positions

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5318 6c8d7289-2bf4-0310-a012-ef5d649a1542
2008-11-03 00:27:23 +00:00
low012
83967f8c77 *) servlet does not forget chosen blacklist anymore when editing, moving or delting an entry
*) move or edit will only be performed if new value actually differs from old one

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5309 6c8d7289-2bf4-0310-a012-ef5d649a1542
2008-10-30 00:03:14 +00:00
low012
04e41a392f *) fixed bug where RegExes were not deleted and even added to the list a second time when the user tried to edit them
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5308 6c8d7289-2bf4-0310-a012-ef5d649a1542
2008-10-29 22:49:44 +00:00
low012
7bac4796d2 *) added servlet which returns all shared blacklists of a peer without information about which part of YaCy (crawler, proxy, ...) blacklist is activated for (to be used for better online import)
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5306 6c8d7289-2bf4-0310-a012-ef5d649a1542
2008-10-27 17:33:43 +00:00
low012
baae3d91b1 *) fixed warning when compiling listManager
*) fixed display of values of information for which part of YaCy (crawler, proxy, ...) blacklist is activated for
*) replaced regular put() with putXML() in several cases

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5305 6c8d7289-2bf4-0310-a012-ef5d649a1542
2008-10-27 16:56:19 +00:00
low012
444575e33d *) prevent XSS when importing blacklist
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5304 6c8d7289-2bf4-0310-a012-ef5d649a1542
2008-10-27 11:06:38 +00:00
low012
a99a629ed4 *) quick fix to prevent comments for blog entries which don't exist (http://forum.yacy-websuche.de/viewtopic.php?f=5&t=1554)
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5302 6c8d7289-2bf4-0310-a012-ef5d649a1542
2008-10-25 12:04:10 +00:00
low012
00e27e5050 *) fixed bug which made it possible to write files outside of the DATA/LIST directory when creating a new blacklist
*) a blacklist will only be created if no blacklist with same name exists (some refactoring has been necessary for this)
*) further minor fixes
*) to be continued...

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5301 6c8d7289-2bf4-0310-a012-ef5d649a1542
2008-10-25 00:11:03 +00:00
orbiter
0edec2b760 FULL redesign of algorithms in htmlTools to encode/decode strings from/to unicode and html.
The old process used a not really efficient way to detect html encoding strings in texts.
All calling methods had been adoped to call the new class in an enhanced way with less parameters.

Many classes in interfaces used a XML encoding only (instead of full html conversion from unicode to html); this behavior was not changed with this commit but should be controlled again since it points out possible XSS leaks

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5295 6c8d7289-2bf4-0310-a012-ef5d649a1542
2008-10-22 18:59:04 +00:00
orbiter
47f0c3b002 replaced the cacheAdmin with the ViewFile servlet, because the cacheAdmin was an interface to the old HTCACHE data structure which does not exist any more. Changed links to point to the ViewFile servlets.
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5289 6c8d7289-2bf4-0310-a012-ef5d649a1542
2008-10-21 11:27:50 +00:00
orbiter
1778fb420d - added some performance tweaks to the new BLOB buffer
- removed the now superfluous HT storage thread
- reduced number of file decompression by shifting the compression moment to the future


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5286 6c8d7289-2bf4-0310-a012-ef5d649a1542
2008-10-19 18:10:42 +00:00
low012
77e41da7d2 *) further propagation of display value (see http://forum.yacy-websuche.de/viewtopic.php?f=5&t=1536)
*) removed another depreciated parameter "time" which led to ugly -UNRESOLVED_PATTERN- in URL

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5285 6c8d7289-2bf4-0310-a012-ef5d649a1542
2008-10-18 19:39:46 +00:00
low012
ff46ce8520 *) fixed display=2 (see http://forum.yacy-websuche.de/viewtopic.php?f=5&t=1536)
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5283 6c8d7289-2bf4-0310-a012-ef5d649a1542
2008-10-17 19:57:39 +00:00
orbiter
826ca79735 refactoring and new architecture to store the files of the web cache:
- files are not stored any more as individual files
- a new database structure using BLOBHeap files stores many cache entries in common files
- all file-writing procedures had been migrated to generate byte[] objects which are written with the new database methods

this is only an intermediate step to the final architecture, where cached files are written together with their metadata in one single database structure.

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5276 6c8d7289-2bf4-0310-a012-ef5d649a1542
2008-10-16 21:24:09 +00:00
low012
2b18a9b2c4 *) removed depreciated parameter "time" which led to ugly -UNRESOLVED_PATTERN- in URL
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5275 6c8d7289-2bf4-0310-a012-ef5d649a1542
2008-10-16 19:31:29 +00:00
orbiter
7860d5d632 fix for bug in seed list management (cause was bad class overloading, only visual effects!)
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5265 6c8d7289-2bf4-0310-a012-ef5d649a1542
2008-10-12 19:51:53 +00:00
lotus
603282bcf4 fix for out of bounds exception
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5264 6c8d7289-2bf4-0310-a012-ef5d649a1542
2008-10-11 07:47:34 +00:00
orbiter
ffed5fc415 fixed problem with lost peers in database
migrated seedDB from BLOBTree to BLOBHeap

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5263 6c8d7289-2bf4-0310-a012-ef5d649a1542
2008-10-10 14:40:02 +00:00
lotus
736dd86193 - option enableSimpleConfig can disable hidden tables
- corrected some Xmx values
- friendlier welcome message format

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5259 6c8d7289-2bf4-0310-a012-ef5d649a1542
2008-10-09 12:48:43 +00:00
orbiter
ff68f394dd fix for problem with balancer and lost crawl profiles:
if crawl profile ist lost, no robots.txt is loaded any more

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5258 6c8d7289-2bf4-0310-a012-ef5d649a1542
2008-10-08 18:26:36 +00:00
apfelmaennchen
3717d2057a YaCy-UI: fix for http://forum.yacy-websuche.de/viewtopic.php?f=5&t=1483
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5251 6c8d7289-2bf4-0310-a012-ef5d649a1542
2008-10-05 18:50:43 +00:00
orbiter
d0bdcdd57c small changes to attributes of DoS attack protection parameters
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5246 6c8d7289-2bf4-0310-a012-ef5d649a1542
2008-10-03 19:44:42 +00:00
orbiter
9ac16f565b - fixed several bugs in database management functions
- fixed a display bug for the performance graph
- fixed deadlock when initialization of awt happens simultanously
- removed some debugging output

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5245 6c8d7289-2bf4-0310-a012-ef5d649a1542
2008-10-03 18:57:02 +00:00
lotus
7fdf65339d system status dropped into next line if seed server was enabled. display needs about 230px, set fixed width again.
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5243 6c8d7289-2bf4-0310-a012-ef5d649a1542
2008-10-03 11:42:24 +00:00
lotus
7782a43060 fix if LANGUAGE: was not defined and the end of the query
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5242 6c8d7289-2bf4-0310-a012-ef5d649a1542
2008-10-03 11:36:17 +00:00
apfelmaennchen
2c23e6ad34 YaCy-UI: added some features for Admin Console
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5241 6c8d7289-2bf4-0310-a012-ef5d649a1542
2008-10-03 05:32:31 +00:00
apfelmaennchen
efcae14714 YaCy-UI:
- added 'Open' button to search result toolbar
- lets you open all selected search results in new window/tab
- added 'any language' filter as default

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5240 6c8d7289-2bf4-0310-a012-ef5d649a1542
2008-10-02 17:38:03 +00:00
lotus
902a0d0f38 fieldset of system status was bigger than defined space. IE overlapped some text.
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5238 6c8d7289-2bf4-0310-a012-ef5d649a1542
2008-10-01 20:33:32 +00:00
orbiter
820a03f9d6 - removed some warnings
- used fix in SVN 5233 for ysearch.java and search.java

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5237 6c8d7289-2bf4-0310-a012-ef5d649a1542
2008-10-01 20:20:39 +00:00
lotus
69925a7e91 fix for http://forum.yacy-websuche.de/viewtopic.php?f=6&t=1441
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5236 6c8d7289-2bf4-0310-a012-ef5d649a1542
2008-10-01 20:16:50 +00:00
lotus
fe2792e9ce use accept-language header instead of user agent for language detection
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5235 6c8d7289-2bf4-0310-a012-ef5d649a1542
2008-10-01 17:47:11 +00:00
lotus
e5904e6a21 removed color definition for input elements in default skin
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5234 6c8d7289-2bf4-0310-a012-ef5d649a1542
2008-10-01 13:07:24 +00:00
lotus
93ddf206e6 opensearch fix if user agent had no language
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5233 6c8d7289-2bf4-0310-a012-ef5d649a1542
2008-09-30 20:13:18 +00:00
lotus
b8538fae04 search form like on result page
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5232 6c8d7289-2bf4-0310-a012-ef5d649a1542
2008-09-30 19:59:55 +00:00
lotus
3cce13d1b7 more compact search form
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5231 6c8d7289-2bf4-0310-a012-ef5d649a1542
2008-09-30 18:03:52 +00:00
lotus
95fddf056c - better support for narrow windows on searchpage
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5230 6c8d7289-2bf4-0310-a012-ef5d649a1542
2008-09-30 17:50:42 +00:00
orbiter
c8bdd965ec - larger update time for status page
- balancer writes cause of robots.txt in log file for crawl delay
- removed log output for forced GC
- smaller RAM flush for RWI cache, should cause more usage of cache and faster crawling

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5228 6c8d7289-2bf4-0310-a012-ef5d649a1542
2008-09-30 11:09:46 +00:00
lotus
3a919bf24e better solution for search result layout
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5227 6c8d7289-2bf4-0310-a012-ef5d649a1542
2008-09-29 19:11:16 +00:00
lotus
f95ec8b813 fix for non-accessible 2nd-line tabs on admin console
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5226 6c8d7289-2bf4-0310-a012-ef5d649a1542
2008-09-29 18:33:38 +00:00
orbiter
c44e97d6dd more lines in log on status page
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5225 6c8d7289-2bf4-0310-a012-ef5d649a1542
2008-09-29 18:07:55 +00:00
orbiter
dc149df3b1 new status page layout:
- smaller kaskelix image to make room for more information
- added the memory graph, since this picture is widely used to monitor YaCys activities
- added border to log line iframe (looks better together with memory graph)

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5224 6c8d7289-2bf4-0310-a012-ef5d649a1542
2008-09-29 14:59:06 +00:00
daburna
298196e7a4 translation update
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5223 6c8d7289-2bf4-0310-a012-ef5d649a1542
2008-09-29 13:24:03 +00:00
lotus
dda771db9d - search result layout
- tray only for windows

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5222 6c8d7289-2bf4-0310-a012-ef5d649a1542
2008-09-29 12:39:57 +00:00
apfelmaennchen
7f88dadf87 YaCy-UI: added a notice to a bug in bookmarks folder recursion!
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5221 6c8d7289-2bf4-0310-a012-ef5d649a1542
2008-09-29 06:02:48 +00:00
orbiter
2126315d40 layout changes for search result page
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5220 6c8d7289-2bf4-0310-a012-ef5d649a1542
2008-09-28 23:19:56 +00:00
low012
f7003aa75f *) updated whos.amung.us code in terminal page
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5217 6c8d7289-2bf4-0310-a012-ef5d649a1542
2008-09-28 12:54:01 +00:00
apfelmaennchen
cc9f1e9dd9 YaCy-UI: enabled edit bookmarks in toolbar
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5215 6c8d7289-2bf4-0310-a012-ef5d649a1542
2008-09-27 21:15:30 +00:00
apfelmaennchen
00c843f7d9 YaCy-UI:
- added scrollbar to folder widget
- slightly improved Ajax handling for folder and tag widget

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5214 6c8d7289-2bf4-0310-a012-ef5d649a1542
2008-09-27 11:38:20 +00:00
lotus
9d4f0325e1 - removed shutdown from search page (we have it in tray now!)
- fixed doubleclick action for tray

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5211 6c8d7289-2bf4-0310-a012-ef5d649a1542
2008-09-26 10:55:08 +00:00
apfelmaennchen
f293554216 XBEL: small bug fix
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5208 6c8d7289-2bf4-0310-a012-ef5d649a1542
2008-09-25 05:26:36 +00:00
apfelmaennchen
848dbbd280 YaCy-UI:
- added bookmark folder widget to sidebar (it uses the xbel.xml with XSLT!

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5206 6c8d7289-2bf4-0310-a012-ef5d649a1542
2008-09-24 21:09:12 +00:00
apfelmaennchen
7462b5db7b /xml/bookmarks/xbel/xbel.xml:
- added support for xsl-style sheets: style=/path-to-xsl/style.xsl
- removed fixed root folder "YaCy Bookmarks"

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5205 6c8d7289-2bf4-0310-a012-ef5d649a1542
2008-09-24 21:04:27 +00:00
orbiter
6e7d113eac fix for wrong index initialization after network switch
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5203 6c8d7289-2bf4-0310-a012-ef5d649a1542
2008-09-23 23:30:25 +00:00
orbiter
7b35d54c6c fixed some problems with network switching (was not completely 'clean')
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5200 6c8d7289-2bf4-0310-a012-ef5d649a1542
2008-09-23 12:11:19 +00:00
apfelmaennchen
8b6a0376db YaCy-UI:
- another XSS fix (TagCloud)
- upload of missing icons for toolbar

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5198 6c8d7289-2bf4-0310-a012-ef5d649a1542
2008-09-22 05:19:37 +00:00
apfelmaennchen
2ff6f0360e YaCy-UI: small fixes to accomodate Safari & Opera
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5196 6c8d7289-2bf4-0310-a012-ef5d649a1542
2008-09-21 14:26:35 +00:00
apfelmaennchen
a28af9d4c4 YaCy-UI: enabled language selection - special thanks to orbiter
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5195 6c8d7289-2bf4-0310-a012-ef5d649a1542
2008-09-21 10:49:05 +00:00
apfelmaennchen
1d9f0234c8 YaCy-UI:
- added basic functionallity to add bookmark(s) from search results
- re-design of sidebar search form
- replaced classic search with compare search 

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5194 6c8d7289-2bf4-0310-a012-ef5d649a1542
2008-09-21 10:04:51 +00:00
orbiter
1198eeecc7 added language selection to search query:
- the language can be selected using a LANGUAGE:<language> element in the query line, i.e.:
java LANGUAGE:en
- the language can be selected with a post element in google-style syntax with the 'rl' element:
?lr=lang_en&query=java

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5193 6c8d7289-2bf4-0310-a012-ef5d649a1542
2008-09-21 07:28:57 +00:00
orbiter
00c1535f84 added ranking and evaluation of language type in a search
the wanted language is taken from the browser user-agent string

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5192 6c8d7289-2bf4-0310-a012-ef5d649a1542
2008-09-21 00:04:42 +00:00
apfelmaennchen
e201ad0e9f YaCy-UI:
- moved search result to flexigrid
- added a toolbar similar to YaCyBar

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5191 6c8d7289-2bf4-0310-a012-ef5d649a1542
2008-09-20 22:21:56 +00:00
apfelmaennchen
06c07f2e0f YaCy-UI: fixed broken style for add bookmark dialog
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5190 6c8d7289-2bf4-0310-a012-ef5d649a1542
2008-09-20 19:33:20 +00:00
apfelmaennchen
a5f61e8b1b YaCy-UI:
- fixed xss vulnurability
- improved add bookmark dialog

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5189 6c8d7289-2bf4-0310-a012-ef5d649a1542
2008-09-20 18:28:58 +00:00
lotus
a81cb78211 finally some putHTML on htroot/xml/
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5188 6c8d7289-2bf4-0310-a012-ef5d649a1542
2008-09-20 07:55:30 +00:00
orbiter
bfcf9b7aa3 - added language detection using metadata from documents: html and odt documents provide this information
- metadata and results from statistical analysis are compared and result is printed out as debug lines
- added ranking profile for wanted language
- added class with ISO 639 table, a list of all valid country codes that will be used for the language identification

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5187 6c8d7289-2bf4-0310-a012-ef5d649a1542
2008-09-19 22:19:11 +00:00
apfelmaennchen
3768a1bd32 set encoding="UTF-8" for getpageinfo_p.xml
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5186 6c8d7289-2bf4-0310-a012-ef5d649a1542
2008-09-19 14:29:10 +00:00
apfelmaennchen
5e8bd0f29c small fixes to getpageinfo_p.xml and htmlFilterContentScraper.java with respect to keyword extraction
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5185 6c8d7289-2bf4-0310-a012-ef5d649a1542
2008-09-19 14:27:44 +00:00
lotus
029e16b653 replaced some put(String, String) by putHTML(String, String) on serverObjects respond
in htroot/ root
didn't touch htroot/xml/
this should solve potential xss issues

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5184 6c8d7289-2bf4-0310-a012-ef5d649a1542
2008-09-19 11:45:11 +00:00
apfelmaennchen
5b2a57bfd0 - /xml/util/getpageinfo_p.xml added <desc> and <lang> tags
- changed htmlFilterContentScraper.getKeywords() to split either space or comma charater not both

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5183 6c8d7289-2bf4-0310-a012-ef5d649a1542
2008-09-18 21:01:23 +00:00
orbiter
ea5de7436d added Sciencenet to the compare search
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5178 6c8d7289-2bf4-0310-a012-ef5d649a1542
2008-09-18 10:56:18 +00:00
orbiter
1bbf362cef update to the crawl balancer: better organization and better crawl delay prediction
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5176 6c8d7289-2bf4-0310-a012-ef5d649a1542
2008-09-17 21:45:21 +00:00
orbiter
ddcf285499 - fixed a bug in performance setting (did not work with german translation)
- reduced maximum number of error url references to save some memory (this was actually a small memory leak)

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5174 6c8d7289-2bf4-0310-a012-ef5d649a1542
2008-09-16 23:04:24 +00:00
orbiter
0cd0fee546 fixed bug with wrong proxy result enqueueing. See:
http://forum.yacy-websuche.de/viewtopic.php?p=8130#p8130
- removed the online status property. This influenced the proxy behavior and created some complexity that was not needed because the online status was never used as it was ceated for (offline browsing)
- checked all proxy identification procedures during crawling and enhanced transparency and error checking
- fixed a proxy identification routine that caused the wrong selection of the proxy result queue

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5173 6c8d7289-2bf4-0310-a012-ef5d649a1542
2008-09-16 21:56:23 +00:00
apfelmaennchen
e071f759d2 YaCy-UI: small optical changes
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5172 6c8d7289-2bf4-0310-a012-ef5d649a1542
2008-09-16 21:39:14 +00:00
apfelmaennchen
bbacf86fe8 - added /xml/bookmarks/posts/add_p.xml
- security fix to /xml/bookmarks/posts/delete_p.xml
- YaCy-UI: added 'add' and 'delete' bookmarks

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5171 6c8d7289-2bf4-0310-a012-ef5d649a1542
2008-09-16 21:38:13 +00:00
apfelmaennchen
cd1ac5bb90 - fixed security issue with /xml/util/ynetSearch.xml
- hopefully fixed YaCy-UI local search with async=false for Ajax-request

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5170 6c8d7289-2bf4-0310-a012-ef5d649a1542
2008-09-16 05:55:31 +00:00
orbiter
c73cf05ddd tried to fix local search in yacy-ui
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5169 6c8d7289-2bf4-0310-a012-ef5d649a1542
2008-09-15 21:56:53 +00:00
apfelmaennchen
99ff478d63 YaCy-UI: small optical fix
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5167 6c8d7289-2bf4-0310-a012-ef5d649a1542
2008-09-15 20:30:33 +00:00
orbiter
7e24c51fd5 - removed alternative search page in main menue in favor of rich client search page
- added necessary option to search request of yacy-ui to get snippets

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5165 6c8d7289-2bf4-0310-a012-ef5d649a1542
2008-09-15 19:00:22 +00:00
apfelmaennchen
7c5867a832 Major update to YaCy-UI
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5162 6c8d7289-2bf4-0310-a012-ef5d649a1542
2008-09-15 17:18:07 +00:00
lotus
73f233bb11 * set resource observer to 1000MB
* transparent favicon

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5160 6c8d7289-2bf4-0310-a012-ef5d649a1542
2008-09-15 12:41:27 +00:00
orbiter
1be24158a2 small fix / rendering option
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5159 6c8d7289-2bf4-0310-a012-ef5d649a1542
2008-09-15 10:22:05 +00:00
orbiter
105be67998 - some bugfixes to compare search
- redesigned input boxes: smaller, more space for result page

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5158 6c8d7289-2bf4-0310-a012-ef5d649a1542
2008-09-15 09:43:51 +00:00
orbiter
693fa2a157 - renamed Comparison to compare_yacy
- added more search engines
- some refactoring and added a list that is used to present the search engine list in a specific order
- added simpleheader and no-header options
- added the compare search to the simple header
- added default compare search page selection storage - after re-start you get the same default search engines as you selected before

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5157 6c8d7289-2bf4-0310-a012-ef5d649a1542
2008-09-15 09:17:05 +00:00
lotus
6216105ca5 small fix
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5150 6c8d7289-2bf4-0310-a012-ef5d649a1542
2008-09-14 18:12:26 +00:00
lotus
5e5178b5e8 please use putHTML to avoid XSS
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5149 6c8d7289-2bf4-0310-a012-ef5d649a1542
2008-09-14 18:08:39 +00:00
low012
b33a6cbb77 *) less disturbing elements in yacy frame
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5148 6c8d7289-2bf4-0310-a012-ef5d649a1542
2008-09-14 17:58:08 +00:00
f1ori
98d902b972 * remember last searchwords in Comparison_p.html
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5147 6c8d7289-2bf4-0310-a012-ef5d649a1542
2008-09-14 15:21:57 +00:00
orbiter
bd45c5a2bc integrated the comparison page into the main menu
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5146 6c8d7289-2bf4-0310-a012-ef5d649a1542
2008-09-14 10:40:01 +00:00
low012
5e0390a24c *) Ooooooooops!
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5145 6c8d7289-2bf4-0310-a012-ef5d649a1542
2008-09-13 17:14:41 +00:00
low012
dc56c35289 *) added page to compare results of 2 search engines
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5144 6c8d7289-2bf4-0310-a012-ef5d649a1542
2008-09-13 16:50:01 +00:00
lotus
880d1a83e2 do not change memory and some non-defined tasks with performance profiles
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5142 6c8d7289-2bf4-0310-a012-ef5d649a1542
2008-09-12 11:54:25 +00:00
lotus
b68d06a6e8 performance settings based on network's remote crawl speed
removed some _pro values from config

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5134 6c8d7289-2bf4-0310-a012-ef5d649a1542
2008-09-10 12:52:17 +00:00
orbiter
bb5c898441 enhancements to localsearch behavior
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5131 6c8d7289-2bf4-0310-a012-ef5d649a1542
2008-09-09 10:24:42 +00:00
orbiter
3c6e8d2015 set default ppm when network is switched
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5127 6c8d7289-2bf4-0310-a012-ef5d649a1542
2008-09-08 18:20:05 +00:00
orbiter
20c2d3c248 fix for bad formatting in CrawlResults
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5126 6c8d7289-2bf4-0310-a012-ef5d649a1542
2008-09-08 13:59:35 +00:00
lotus
b92105c8b0 do not change auto recrawl scheduler with performance profiles
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5123 6c8d7289-2bf4-0310-a012-ef5d649a1542
2008-09-07 13:59:24 +00:00
orbiter
d09ddabd09 corrected a design mistake (5-byte hashes not necessary)
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5119 6c8d7289-2bf4-0310-a012-ef5d649a1542
2008-09-04 21:28:00 +00:00
orbiter
c97d0fcee7 modified the domain list export function:
- used the new superfast domain list generation from the domain statistics
- better interactive behavior

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5118 6c8d7289-2bf4-0310-a012-ef5d649a1542
2008-09-04 20:28:36 +00:00
orbiter
77ee0765a4 - added domain statistic generation to IndexControlURLs_p.html servlet
- added 'delete all' button to all results of such a domain statistic output which causes that all urls to this domain are deleted
- extended stack cleaner to clean also the statistics: they are not completely destroyed, only the smallest counting domains are removed


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5117 6c8d7289-2bf4-0310-a012-ef5d649a1542
2008-09-04 19:41:57 +00:00
lotus
44bc8311af translation fix
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5116 6c8d7289-2bf4-0310-a012-ef5d649a1542
2008-09-04 19:26:59 +00:00
lotus
e5c0b969d6 * save performance profile speed
* fix for wrong javastart_priority after first start

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5115 6c8d7289-2bf4-0310-a012-ef5d649a1542
2008-09-04 19:12:22 +00:00
lotus
d7a16c1f30 * added shutdown on search page (this page is shown after clicking the tray icon)
* shorter, less technical words for configuration-links

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5114 6c8d7289-2bf4-0310-a012-ef5d649a1542
2008-09-04 12:51:05 +00:00