Marown/yacy_search_server
1
0
Fork 0
mirror of https://github.com/yacy/yacy_search_server.git synced 2024-09-21 00:00:13 +02:00
Code Issues Actions Packages Projects Releases Wiki Activity
4,531 Commits 1 Branch 20 Tags 836 MiB
f06c046f6c
Commit Graph

4531 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
orbiter
8fe39ebd74 -fixed file transmission with POST. The only usage was in ranking transmission, therefore: 
-fixed ranking transmission

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4681 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-04-12 08:12:51 +00:00
orbiter
82a9861779 fix for last commit 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4680 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-04-11 12:55:43 +00:00
orbiter
5d1fbb25e7 fix for bad deploy: 
- the name of downloaded release files is adopted if the httpc delivers uncompressed tar.gz files (the .gz is removed from the file name)
- the deploy method is able to handle tar-file (not tar.gz-files)

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4679 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-04-11 12:37:17 +00:00
orbiter
202a3adb3e refactoring of HttpClient Writer processes 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4678 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-04-10 22:47:05 +00:00
danielr
8aa9fd8f24 HTTPC with only 1 retry 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4677 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-04-10 16:47:57 +00:00
orbiter
444dce7e81 more performance hacks 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4676 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-04-10 15:28:58 +00:00
orbiter
2c2dcd12a2 - enhanced performance of Eco-Tables: less time-consuming size() - operations 
- will increase speed of indexing and collection.index creation


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4675 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-04-10 13:24:55 +00:00
orbiter
e356625b22 - refacotring of stream copy handling to support time-consuming operations 
- made usage of BufferedStreams explizit to distinct different copy method in serverFileUtils (byte-by-byte and using an own buffer)
- introduced another timeout setting (java internal property)
- more restrictions to clients accessing a single host (a security setting to prevent DoS by mistake)

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4674 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-04-10 09:53:07 +00:00
danielr
f01c50cf8d Proxy logging error (first step to resolution!?) 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4673 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-04-10 06:56:06 +00:00
orbiter
c3342e1178 - removed class with only one static method 
- removed connection method with too long time-out

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4672 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-04-09 23:35:20 +00:00
orbiter
f97971b63b fixed NPE problems doing a shutdown from command-line 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4671 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-04-09 22:59:17 +00:00
danielr
7a35126e91 http timeouts von alten httpc wieder gesetzt 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4670 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-04-09 11:02:14 +00:00
orbiter
2c1c3bb6eb - some refactoring (sorry Daniel, hab in deinem Code rumgewütet) 
- fixed broken downloads (flush was missing)
- different problem handling when download is corrupted
- different default values in yacy.init

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4669 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-04-08 21:36:33 +00:00
danielr
d96e2badc7 - fixed POST in proxy 
- prepared http connection tracking
- refactoring (mainly moving StreamTools to serverFileUtils)


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4668 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-04-08 21:17:40 +00:00
orbiter
14404d31a8 - enhanced performance graph (more info) 
- added conditions for rarely used logging lines to prevent unnecessary CPU usage for non-printed info

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4667 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-04-08 14:44:39 +00:00
orbiter
696b8ee3f5 fix for http://forum.yacy-websuche.de/viewtopic.php?p=6806#p6806 
- removed all InputStream.available() because this does not work for files > 2GB
- iterator terminate when a IOException occurs
- added handling of non-executing index.add methods to enhance assert usage
- added index for file indexes > 2GB, to be used in new indexHeap

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4666 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-04-08 11:55:59 +00:00
danielr
94d3d3a86f fixed Proxy (for GET, POST still does not work!) 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4665 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-04-08 09:34:20 +00:00
danielr
081ed1d3ec HTTPLoader: reduced stackTraces 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4664 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-04-07 16:56:15 +00:00
danielr
8b2efb6f8c fixed garbage in HTCACHE 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4663 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-04-07 16:46:45 +00:00
orbiter
225f9fd429 various fixes 
- shutdown behavior (killing of client sessions)
- EcoFS reading better
- another synchronization in balancer.size()


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4662 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-04-07 13:12:58 +00:00
orbiter
6e36c156e8 added more logging to EcoFS 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4661 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-04-07 09:52:25 +00:00
danielr
fb541f9162 HTTPC: default timeout half-hour 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4660 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-04-07 09:48:49 +00:00
danielr
a94f6cdca4 HTTPC: allowed self-signed certs 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4659 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-04-07 09:21:43 +00:00
danielr
ab330cfdca Network.html: removed ; from location 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4658 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-04-07 08:13:38 +00:00
orbiter
319144f4b2 fix for outofbounds-excception in EcoFS chunk iterator 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4657 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-04-06 22:28:17 +00:00
danielr
41e9c5723c try to fix shown location (instead of 'Europe/de) JakartaHttpClient/3.') 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4656 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-04-06 22:17:09 +00:00
orbiter
ac8592a102 eclipse build path update 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4655 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-04-06 20:35:05 +00:00
orbiter
a9cf6cf2f4 generalization of index container-heap class. 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4654 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-04-06 20:31:16 +00:00
orbiter
f099061944 protection against bad dht-flush word selection 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4653 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-04-06 20:25:05 +00:00
orbiter
5e4fddc1e6 more logging for new EcoFS.ChunkIterator to find bug for 
http://forum.yacy-websuche.de/viewtopic.php?f=6&t=1024&hilit=&p=6806#p6806

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4652 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-04-06 18:47:49 +00:00
low012
4894df71ab *) moved commons-httpclient from libx to lib (seems to work on my system, I hope one everybody else's too) 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4651 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-04-06 18:24:38 +00:00
orbiter
117ae78001 speed enhancement for reading of eco-table indexes 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4647 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-04-06 11:50:15 +00:00
low012
e96ecd269f *) hopefully fixed build script (included commons-httpclient) 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4645 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-04-06 09:49:45 +00:00
danielr
7c149a4ee8 - undo less 'binary data found' 
- removed duplicate stackTrace


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4643 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-04-05 17:46:11 +00:00
danielr
96cce8bed9 reduced 'Binary data found' errors 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4642 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-04-05 14:20:01 +00:00
danielr
2aef1414f5 removed test (in yacy.init) 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4641 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-04-05 13:49:25 +00:00
danielr
5c3c1fdf41 replaced httpc with Apache Jakarta Commons HttpClient (includes some refactoring ;) 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4640 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-04-05 13:17:16 +00:00
orbiter
daa04f5db9 added additional check in file handler to prevent that url attacks are hidden in url path encodings 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4637 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-04-04 12:15:27 +00:00
orbiter
783a4c9edb strong speed enhancements for the index cache dump and restore: 
storage and loading is 30 times faster! a cache of 100000 RWIs needed 180 seconds
to store and 100 seconds to restore; now the same cache needs only 6 seconds to store and
3 seconds to restore. The cache size has decreased now by 30% (95 MB instead of 150 MB).

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4634 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-04-02 13:18:23 +00:00
orbiter
442204a1c8 fix for concurrentModificationException 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4633 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-04-01 21:21:37 +00:00
orbiter
d2f4926951 - more logging for balancer to get a hint where the problem is 
- fix for new concurrency method in kelondroSplitTable

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4631 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-03-31 18:45:27 +00:00
orbiter
20dadba426 - added a deadlock prevention function in cache flushing 
- removed unused methods in collection index

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4630 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-03-31 17:51:51 +00:00
orbiter
764a40e37d speed enhancements for crawler and url retrieval (affects also search speed) 
- concurrency for LURL-fetching: this can be done using a concurrent lookup into the separated url databases. Concurrency is possible because there is no IO during lookup. The more LURL-Tables are present, the better is the speedup. More CPUs will increase speed
- because a large number of LURL-lookups are made during crawling (for double-check), the LURL-Lookup speed enhancements enhances also crawling speed
- search speed also profits from LURL-lookup enhancement
- changed some flushing parameters in word index caching which should make better use of large word index caches and should speed up indexing
- removed flush chunksize parameter, because this was only useful for IO path enhancement feature which was removed some weeks ago to prevent blocking and deadlocks during search requests

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4628 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-03-31 15:41:19 +00:00
orbiter
3ce3a4a3a1 added stub for new index container heap data structure (purpose: index folding) 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4627 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-03-30 22:58:42 +00:00
orbiter
2c34038912 addition/correction to last commit: usage of concurrent-classes 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4626 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-03-30 21:17:12 +00:00
orbiter
b2150057d2 removed unnecessary cleanup method 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4625 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-03-30 20:32:08 +00:00
f1ori
76eac114ed * define global javascript-variable with var to get rid of warnings 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4624 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-03-30 19:51:19 +00:00
low012
b63cf2fc1c *) added button to Crawl Profile Editor to delete all terminated crawl jobs (only visible if there are terminated crawl jobs) 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4620 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-03-30 15:15:56 +00:00
f1ori
2aed6bb3f7 * return valid xml in xml-bookmarks 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4619 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-03-30 14:58:29 +00:00
lulabad
c4c0d54b22 * added regex extended blacklistengine 
* removed my own engines

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4618 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-03-30 08:50:09 +00:00