Marown/yacy_search_server
1
0
Fork 0
mirror of https://github.com/yacy/yacy_search_server.git synced 2024-09-21 00:00:13 +02:00
Code Issues Actions Packages Projects Releases Wiki Activity
5,095 Commits 1 Branch 20 Tags 836 MiB
eab72424df
Commit Graph

5095 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
low012
eab72424df *) Fixed small bug: When adding new elements to blacklist via import, the blacklist which the elements were added to was supposed to be displayed, which did not work correctly. 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5414 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-12-28 09:58:02 +00:00
low012
0e56675596 *) cleaning up ;-) 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5413 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-12-27 20:09:36 +00:00
low012
cf69557ea2 *) blacklists can be exported as XML or plain text now 
*) blacklist import via file upload works now

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5412 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-12-27 15:38:20 +00:00
low012
1594a15be9 *) explicit mentioning of blacklist in blacklist cleaner 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5411 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-12-27 13:06:05 +00:00
f1ori
2d2ce24011 * remove all encoding-stuff from proxy 
encoding is handled by parsers or browser, proxy only passes through


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5410 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-12-23 19:14:54 +00:00
f1ori
73c8a0839c * abort download, when proxy connection is closed 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5409 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-12-23 11:30:24 +00:00
orbiter
bb935fdbb0 less organization overhead for DNS caching and prefetching 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5408 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-12-23 10:06:49 +00:00
f1ori
4907697cfa * make fileuploads through proxy bigger than 65500 bytes possible 
* remove gzip-encoding for files from cache


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5407 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-12-22 23:04:00 +00:00
orbiter
fc8189f3fb better self-healing of corrupted databases 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5406 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-12-22 16:43:49 +00:00
f1ori
963da8c3f9 * updated tm-extractors to new version 1.0 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5405 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-12-21 14:51:03 +00:00
f1ori
51f1a1927c * remove saaj.jar and axis.jar and references to it (was for soap-stuff?) 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5404 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-12-21 13:06:04 +00:00
low012
5a89266598 *) new parameters for future use (better blacklist handling for im- and export) 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5403 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-12-19 19:33:08 +00:00
orbiter
e34ac22fbd - added new monitoring servlet at 
http://localhost:8080/PerformanceConcurrency_p.html
- used the new monitoring to do some fine-tuning of the indexing queue

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5402 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-12-19 15:26:01 +00:00
lotus
449e697436 fix for null-seed in seedfile 
http://forum.yacy-websuche.de/viewtopic.php?f=5&t=1653

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5401 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-12-19 12:10:01 +00:00
orbiter
d376d81fc4 replaced busy thread control of crawl stacker by blocking threads 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5400 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-12-18 23:18:34 +00:00
orbiter
f29b48d9ff patch for IndexOutOfBoundsException 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5399 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-12-18 22:05:26 +00:00
f1ori
0881190b19 * Robots.txt: don't interpret Crawl-Delays for other robots 
fixes: http://forum.yacy-websuche.de/viewtopic.php?f=5&t=1647


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5398 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-12-18 15:35:41 +00:00
orbiter
243e73f53b removed unnecessary usage of kelondroBLOBTree 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5397 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-12-18 00:18:37 +00:00
orbiter
8cb7170b75 - set status of kelondroTree, kelondroBLOBTree and kelondroFlexTable to deprecated 
- removed initialization and/or usage of kelondroFlexTable (should meanwhile not be used any more)

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5396 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-12-18 00:08:17 +00:00
orbiter
7535fd7447 - refactoring of CrawlEntry and CrawlStacker 
- introduced blocking queues in CrawlStacker to make it ready for concurrency
- added a second busy thread for the CrawlStacker
The CrawlStacker is multithreaded. It shall be transformed into a BlockingThread in another step.
The concurrency of the stacker will hopefully solve some problems with cases where DNS blocks.

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5395 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-12-17 22:53:06 +00:00
lotus
6569cbbec1 npe fix: http://forum.yacy-websuche.de/viewtopic.php?t=1646 
(break to avoid bad side effects)

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5394 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-12-16 20:53:31 +00:00
lotus
18513e2ee2 npe fix: http://forum.yacy-websuche.de/viewtopic.php?t=1646 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5393 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-12-16 13:36:13 +00:00
orbiter
2802138787 - refactoring of CrawlStacker (to prepare it for new multi-Threading to remove DNS lookup bottleneck) 
- fix of shallBeOwnWord target computation heuristic


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5392 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-12-15 00:02:58 +00:00
lotus
b1e211b258 no error-alert: http://forum.yacy-websuche.de/viewtopic.php?t=1639 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5391 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-12-13 12:04:08 +00:00
orbiter
13cb0916ee changes to statistics and content of thread dump servlet 
(points now more directly to performance leaks without mentioning class calls inside of sun/java calls that cannot be changed anyway)

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5390 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-12-11 20:13:14 +00:00
orbiter
db6b3bf5a3 speed enhancement for integrated http server: 
- tuning hacks in template engine
- bypassing the template engine if no servlet present

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5389 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-12-11 20:10:37 +00:00
orbiter
7cd08bd5fb fix for NPE in BLOBCompressor 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5388 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-12-11 13:33:24 +00:00
orbiter
5b94498643 fine-tuning of cache usage from SVN 5386 and a bug fix for overflow in available() method 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5387 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-12-10 14:35:01 +00:00
orbiter
1779c3c507 - added a read cache to the RAFile interface to RandomAccessFile 
- added a write buffer to BLOBHeap
- modified the BLOBBuffer (is now only to buffer non-compressed content)
- added content compression to the HTCache
The new read cache will decrease the start/initialization time of BLOB files,
like the HTCache, RobotsTxt and other BLOBHeap structures.


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5386 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-12-10 11:15:19 +00:00
orbiter
e1acdb952c fix for problem with userDB and bookmarksDB which was caused by changes in kelondroRA in SVN 5376 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5385 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-12-08 00:17:45 +00:00
lotus
2c682d649b - no stop shortcut (-> stop via tray) 
- store registry keys on current profile

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5384 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-12-07 19:37:49 +00:00
lotus
e918d64c23 show hand-cursor an labels 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5383 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-12-06 17:32:53 +00:00
orbiter
4a2dac659e more speed hacks: 
- modified and activated write buffer
- increased cache flush factor
- fixed a problem with deadlocking of indexing process

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5382 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-12-05 13:55:48 +00:00
lotus
07d7653de1 update to JRE 6u11 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5381 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-12-05 11:23:01 +00:00
lotus
1fb518a5b4 display <String> etc. 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5380 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-12-04 20:21:53 +00:00
orbiter
47292e696a more performance hacks 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5379 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-12-04 12:54:16 +00:00
orbiter
759cef23dd fix for bug in kelondroAbstractRA.readFully 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5378 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-12-03 23:32:07 +00:00
orbiter
bd1dc9cd5d thread dump with statistics, a little bit of profiling 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5377 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-12-03 23:26:25 +00:00
orbiter
d39d420b39 performance hacks 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5376 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-12-03 15:38:29 +00:00
lotus
5280ad638d added basic performance page 
other performance settings can be found on advanced settings

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5375 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-12-03 14:10:01 +00:00
lotus
1a51d9fcfd display proper values 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5374 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-12-02 17:57:30 +00:00
orbiter
0b4808ba3d added new interactive search feature: 
- during the user types search queries, the local database is searched
- results are presented interactively

This was implemented using a new JSON result format for search results in YaCy
- added JSON as file format for servlets
- refactoring of current search servlets (xml and html)
- added JSON output format for search results
- added AJAX-based search page, that uses the yacysearch.json selrvlet to print results as a query is typed

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5373 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-12-02 15:24:25 +00:00
orbiter
74a3d86114 fixed a error response that might present classified information 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5372 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-12-01 23:14:42 +00:00
orbiter
c6525ab75f fix for NPE in seed handling 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5371 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-12-01 23:08:27 +00:00
lotus
fea82b54ef more contrast on search snippets 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5370 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-11-26 19:57:13 +00:00
lotus
1951d30a62 addendum to last commit 
handle words with length < 3 correctly

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5369 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-11-26 19:43:40 +00:00
lotus
325ba7bfb8 only query words with length > 2 
this is not complete, yet

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5368 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-11-26 16:41:38 +00:00
lotus
489edb4473 improved pattern selection 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5367 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-11-26 10:06:38 +00:00
low012
e423fa9846 *) added method to only get file names in directory listing which match a filter 
*) only files which end with .black will be listed as blacklists
*) added a little bit of Javadoc

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5366 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-11-25 20:26:06 +00:00
lotus
577b53aee6 added more search engines 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5365 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-11-24 13:05:20 +00:00