Marown/yacy_search_server
1
0
Fork 0
mirror of https://github.com/yacy/yacy_search_server.git synced 2024-09-21 00:00:13 +02:00
Code Issues Actions Packages Projects Releases Wiki Activity
4,519 Commits 1 Branch 20 Tags 836 MiB
d9d1c8de70
Commit Graph

4519 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
orbiter
2c1c3bb6eb - some refactoring (sorry Daniel, hab in deinem Code rumgewütet) 
- fixed broken downloads (flush was missing)
- different problem handling when download is corrupted
- different default values in yacy.init

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4669 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-04-08 21:36:33 +00:00
danielr
d96e2badc7 - fixed POST in proxy 
- prepared http connection tracking
- refactoring (mainly moving StreamTools to serverFileUtils)


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4668 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-04-08 21:17:40 +00:00
orbiter
14404d31a8 - enhanced performance graph (more info) 
- added conditions for rarely used logging lines to prevent unnecessary CPU usage for non-printed info

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4667 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-04-08 14:44:39 +00:00
orbiter
696b8ee3f5 fix for http://forum.yacy-websuche.de/viewtopic.php?p=6806#p6806 
- removed all InputStream.available() because this does not work for files > 2GB
- iterator terminate when a IOException occurs
- added handling of non-executing index.add methods to enhance assert usage
- added index for file indexes > 2GB, to be used in new indexHeap

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4666 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-04-08 11:55:59 +00:00
danielr
94d3d3a86f fixed Proxy (for GET, POST still does not work!) 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4665 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-04-08 09:34:20 +00:00
danielr
081ed1d3ec HTTPLoader: reduced stackTraces 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4664 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-04-07 16:56:15 +00:00
danielr
8b2efb6f8c fixed garbage in HTCACHE 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4663 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-04-07 16:46:45 +00:00
orbiter
225f9fd429 various fixes 
- shutdown behavior (killing of client sessions)
- EcoFS reading better
- another synchronization in balancer.size()


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4662 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-04-07 13:12:58 +00:00
orbiter
6e36c156e8 added more logging to EcoFS 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4661 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-04-07 09:52:25 +00:00
danielr
fb541f9162 HTTPC: default timeout half-hour 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4660 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-04-07 09:48:49 +00:00
danielr
a94f6cdca4 HTTPC: allowed self-signed certs 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4659 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-04-07 09:21:43 +00:00
danielr
ab330cfdca Network.html: removed ; from location 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4658 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-04-07 08:13:38 +00:00
orbiter
319144f4b2 fix for outofbounds-excception in EcoFS chunk iterator 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4657 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-04-06 22:28:17 +00:00
danielr
41e9c5723c try to fix shown location (instead of 'Europe/de) JakartaHttpClient/3.') 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4656 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-04-06 22:17:09 +00:00
orbiter
ac8592a102 eclipse build path update 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4655 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-04-06 20:35:05 +00:00
orbiter
a9cf6cf2f4 generalization of index container-heap class. 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4654 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-04-06 20:31:16 +00:00
orbiter
f099061944 protection against bad dht-flush word selection 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4653 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-04-06 20:25:05 +00:00
orbiter
5e4fddc1e6 more logging for new EcoFS.ChunkIterator to find bug for 
http://forum.yacy-websuche.de/viewtopic.php?f=6&t=1024&hilit=&p=6806#p6806

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4652 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-04-06 18:47:49 +00:00
low012
4894df71ab *) moved commons-httpclient from libx to lib (seems to work on my system, I hope one everybody else's too) 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4651 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-04-06 18:24:38 +00:00
orbiter
117ae78001 speed enhancement for reading of eco-table indexes 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4647 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-04-06 11:50:15 +00:00
low012
e96ecd269f *) hopefully fixed build script (included commons-httpclient) 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4645 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-04-06 09:49:45 +00:00
danielr
7c149a4ee8 - undo less 'binary data found' 
- removed duplicate stackTrace


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4643 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-04-05 17:46:11 +00:00
danielr
96cce8bed9 reduced 'Binary data found' errors 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4642 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-04-05 14:20:01 +00:00
danielr
2aef1414f5 removed test (in yacy.init) 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4641 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-04-05 13:49:25 +00:00
danielr
5c3c1fdf41 replaced httpc with Apache Jakarta Commons HttpClient (includes some refactoring ;) 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4640 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-04-05 13:17:16 +00:00
orbiter
daa04f5db9 added additional check in file handler to prevent that url attacks are hidden in url path encodings 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4637 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-04-04 12:15:27 +00:00
orbiter
783a4c9edb strong speed enhancements for the index cache dump and restore: 
storage and loading is 30 times faster! a cache of 100000 RWIs needed 180 seconds
to store and 100 seconds to restore; now the same cache needs only 6 seconds to store and
3 seconds to restore. The cache size has decreased now by 30% (95 MB instead of 150 MB).

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4634 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-04-02 13:18:23 +00:00
orbiter
442204a1c8 fix for concurrentModificationException 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4633 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-04-01 21:21:37 +00:00
orbiter
d2f4926951 - more logging for balancer to get a hint where the problem is 
- fix for new concurrency method in kelondroSplitTable

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4631 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-03-31 18:45:27 +00:00
orbiter
20dadba426 - added a deadlock prevention function in cache flushing 
- removed unused methods in collection index

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4630 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-03-31 17:51:51 +00:00
orbiter
764a40e37d speed enhancements for crawler and url retrieval (affects also search speed) 
- concurrency for LURL-fetching: this can be done using a concurrent lookup into the separated url databases. Concurrency is possible because there is no IO during lookup. The more LURL-Tables are present, the better is the speedup. More CPUs will increase speed
- because a large number of LURL-lookups are made during crawling (for double-check), the LURL-Lookup speed enhancements enhances also crawling speed
- search speed also profits from LURL-lookup enhancement
- changed some flushing parameters in word index caching which should make better use of large word index caches and should speed up indexing
- removed flush chunksize parameter, because this was only useful for IO path enhancement feature which was removed some weeks ago to prevent blocking and deadlocks during search requests

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4628 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-03-31 15:41:19 +00:00
orbiter
3ce3a4a3a1 added stub for new index container heap data structure (purpose: index folding) 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4627 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-03-30 22:58:42 +00:00
orbiter
2c34038912 addition/correction to last commit: usage of concurrent-classes 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4626 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-03-30 21:17:12 +00:00
orbiter
b2150057d2 removed unnecessary cleanup method 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4625 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-03-30 20:32:08 +00:00
f1ori
76eac114ed * define global javascript-variable with var to get rid of warnings 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4624 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-03-30 19:51:19 +00:00
low012
b63cf2fc1c *) added button to Crawl Profile Editor to delete all terminated crawl jobs (only visible if there are terminated crawl jobs) 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4620 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-03-30 15:15:56 +00:00
f1ori
2aed6bb3f7 * return valid xml in xml-bookmarks 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4619 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-03-30 14:58:29 +00:00
lulabad
c4c0d54b22 * added regex extended blacklistengine 
* removed my own engines

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4618 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-03-30 08:50:09 +00:00
orbiter
368593e449 enhanced the concurrency handling of indexing process (better queue size control, better data concept, better shutdown behavior) 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4617 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-03-30 00:03:44 +00:00
low012
4c3f1b67ad *) refactoring of Blacklist_p.java (moving entries might be slightly slower, but the code is more tidy now) 
*) added edit functionality for blacklist entries

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4616 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-03-29 20:39:46 +00:00
f1ori
466d49e90c * added login-parameter to be able to force authentication 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4613 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-03-29 11:10:04 +00:00
orbiter
be58135b3e possible fix for deadlock in search execution 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4612 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-03-29 07:50:37 +00:00
f1ori
c67350f138 * use putXML with forXML-parameter to ensure urls are valid xml (problem was & in url) 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4611 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-03-28 22:50:33 +00:00
f1ori
98a03249f3 - optimize selectall-javascript-function (in firefox more then 10 times faster now) 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4610 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-03-28 16:28:36 +00:00
orbiter
0241d070bc added concurrency to indexing process: 
- the methods {parsing, semantic analysis (condensing), structure analysis (web structure)} in the serialized indexing path had been made concurrent.
- four BlockingQueues handle concurrency and hand-over of the indexing objects, the last object in the queue is stored into a blockingQueue of maximum size 1 to serialize the process for storage (which uses IO and therefore here should not be deserialized)
- a concurrency of (CPUs + 1) is default. Single-CPU users will profil from the change because large files cannot block the indexing process any more.
- removed the secondary indexing thread, which is superfluous now. Concurrency is default for all users.

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4609 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-03-28 11:56:28 +00:00
lulabad
9fb5d661f2 added my Blacklistengines 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4608 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-03-28 08:18:21 +00:00
orbiter
bca87f1e38 - refactoring of serverThreads: renaming to distinguish busy-threads and blocking-threads 
- added blockingThreads which are threads that are not driven by pause times but by BlockingQueue lookup

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4606 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-03-27 12:03:16 +00:00
orbiter
968c775025 - preparation of parsing/indexing queue for concurrent execution 
- remote crawl receipts are now transmitted concurrently in separate threads (makes remove crawls much faster!)

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4605 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-03-26 22:43:38 +00:00
orbiter
9b0e20fb06 next refactoring step in document indexing to prepare concurrency environment for document parsing 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4604 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-03-26 19:51:05 +00:00
orbiter
7f9f639d20 - refactoring and abstraction of index reference (urls) handling: blacklisting is part of reference filtering 
- refactoring of word/phrase handling: word abstraction from condenser becomes part of index element handling
- removed unused code parts from condenser

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4603 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-03-26 15:37:49 +00:00