Marown/yacy_search_server
1
0
Fork 0
mirror of https://github.com/yacy/yacy_search_server.git synced 2024-09-21 00:00:13 +02:00
Code Issues Actions Packages Projects Releases Wiki Activity
4,352 Commits 1 Branch 20 Tags 836 MiB
5e4fddc1e6
Commit Graph

4352 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
orbiter
5e4fddc1e6 more logging for new EcoFS.ChunkIterator to find bug for 
http://forum.yacy-websuche.de/viewtopic.php?f=6&t=1024&hilit=&p=6806#p6806

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4652 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-04-06 18:47:49 +00:00
low012
4894df71ab *) moved commons-httpclient from libx to lib (seems to work on my system, I hope one everybody else's too) 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4651 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-04-06 18:24:38 +00:00
orbiter
117ae78001 speed enhancement for reading of eco-table indexes 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4647 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-04-06 11:50:15 +00:00
low012
e96ecd269f *) hopefully fixed build script (included commons-httpclient) 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4645 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-04-06 09:49:45 +00:00
danielr
7c149a4ee8 - undo less 'binary data found' 
- removed duplicate stackTrace


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4643 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-04-05 17:46:11 +00:00
danielr
96cce8bed9 reduced 'Binary data found' errors 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4642 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-04-05 14:20:01 +00:00
danielr
2aef1414f5 removed test (in yacy.init) 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4641 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-04-05 13:49:25 +00:00
danielr
5c3c1fdf41 replaced httpc with Apache Jakarta Commons HttpClient (includes some refactoring ;) 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4640 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-04-05 13:17:16 +00:00
orbiter
daa04f5db9 added additional check in file handler to prevent that url attacks are hidden in url path encodings 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4637 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-04-04 12:15:27 +00:00
orbiter
783a4c9edb strong speed enhancements for the index cache dump and restore: 
storage and loading is 30 times faster! a cache of 100000 RWIs needed 180 seconds
to store and 100 seconds to restore; now the same cache needs only 6 seconds to store and
3 seconds to restore. The cache size has decreased now by 30% (95 MB instead of 150 MB).

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4634 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-04-02 13:18:23 +00:00
orbiter
442204a1c8 fix for concurrentModificationException 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4633 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-04-01 21:21:37 +00:00
orbiter
d2f4926951 - more logging for balancer to get a hint where the problem is 
- fix for new concurrency method in kelondroSplitTable

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4631 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-03-31 18:45:27 +00:00
orbiter
20dadba426 - added a deadlock prevention function in cache flushing 
- removed unused methods in collection index

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4630 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-03-31 17:51:51 +00:00
orbiter
764a40e37d speed enhancements for crawler and url retrieval (affects also search speed) 
- concurrency for LURL-fetching: this can be done using a concurrent lookup into the separated url databases. Concurrency is possible because there is no IO during lookup. The more LURL-Tables are present, the better is the speedup. More CPUs will increase speed
- because a large number of LURL-lookups are made during crawling (for double-check), the LURL-Lookup speed enhancements enhances also crawling speed
- search speed also profits from LURL-lookup enhancement
- changed some flushing parameters in word index caching which should make better use of large word index caches and should speed up indexing
- removed flush chunksize parameter, because this was only useful for IO path enhancement feature which was removed some weeks ago to prevent blocking and deadlocks during search requests

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4628 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-03-31 15:41:19 +00:00
orbiter
3ce3a4a3a1 added stub for new index container heap data structure (purpose: index folding) 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4627 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-03-30 22:58:42 +00:00
orbiter
2c34038912 addition/correction to last commit: usage of concurrent-classes 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4626 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-03-30 21:17:12 +00:00
orbiter
b2150057d2 removed unnecessary cleanup method 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4625 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-03-30 20:32:08 +00:00
f1ori
76eac114ed * define global javascript-variable with var to get rid of warnings 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4624 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-03-30 19:51:19 +00:00
low012
b63cf2fc1c *) added button to Crawl Profile Editor to delete all terminated crawl jobs (only visible if there are terminated crawl jobs) 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4620 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-03-30 15:15:56 +00:00
f1ori
2aed6bb3f7 * return valid xml in xml-bookmarks 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4619 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-03-30 14:58:29 +00:00
lulabad
c4c0d54b22 * added regex extended blacklistengine 
* removed my own engines

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4618 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-03-30 08:50:09 +00:00
orbiter
368593e449 enhanced the concurrency handling of indexing process (better queue size control, better data concept, better shutdown behavior) 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4617 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-03-30 00:03:44 +00:00
low012
4c3f1b67ad *) refactoring of Blacklist_p.java (moving entries might be slightly slower, but the code is more tidy now) 
*) added edit functionality for blacklist entries

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4616 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-03-29 20:39:46 +00:00
f1ori
466d49e90c * added login-parameter to be able to force authentication 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4613 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-03-29 11:10:04 +00:00
orbiter
be58135b3e possible fix for deadlock in search execution 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4612 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-03-29 07:50:37 +00:00
f1ori
c67350f138 * use putXML with forXML-parameter to ensure urls are valid xml (problem was & in url) 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4611 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-03-28 22:50:33 +00:00
f1ori
98a03249f3 - optimize selectall-javascript-function (in firefox more then 10 times faster now) 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4610 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-03-28 16:28:36 +00:00
orbiter
0241d070bc added concurrency to indexing process: 
- the methods {parsing, semantic analysis (condensing), structure analysis (web structure)} in the serialized indexing path had been made concurrent.
- four BlockingQueues handle concurrency and hand-over of the indexing objects, the last object in the queue is stored into a blockingQueue of maximum size 1 to serialize the process for storage (which uses IO and therefore here should not be deserialized)
- a concurrency of (CPUs + 1) is default. Single-CPU users will profil from the change because large files cannot block the indexing process any more.
- removed the secondary indexing thread, which is superfluous now. Concurrency is default for all users.

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4609 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-03-28 11:56:28 +00:00
lulabad
9fb5d661f2 added my Blacklistengines 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4608 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-03-28 08:18:21 +00:00
orbiter
bca87f1e38 - refactoring of serverThreads: renaming to distinguish busy-threads and blocking-threads 
- added blockingThreads which are threads that are not driven by pause times but by BlockingQueue lookup

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4606 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-03-27 12:03:16 +00:00
orbiter
968c775025 - preparation of parsing/indexing queue for concurrent execution 
- remote crawl receipts are now transmitted concurrently in separate threads (makes remove crawls much faster!)

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4605 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-03-26 22:43:38 +00:00
orbiter
9b0e20fb06 next refactoring step in document indexing to prepare concurrency environment for document parsing 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4604 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-03-26 19:51:05 +00:00
orbiter
7f9f639d20 - refactoring and abstraction of index reference (urls) handling: blacklisting is part of reference filtering 
- refactoring of word/phrase handling: word abstraction from condenser becomes part of index element handling
- removed unused code parts from condenser

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4603 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-03-26 15:37:49 +00:00
orbiter
d6050b9ffb - separated the LURL data storage and Crawl result stack for process supervision. 
this is another step to enable multiple, concurrent fulltext-indexes
- another try to make the yacy-httpc more stable

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4602 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-03-26 14:13:05 +00:00
low012
8e889de50b *) Added Lotus' patch (http://forum.yacy-websuche.de/viewtopic.php?t=979), user will be taken back to last opened page after making changes in Advanced Settings. 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4601 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-03-25 20:40:54 +00:00
orbiter
8d6a13bc07 refactoring of parsing-condensing-indexing process: 
- separated parts
- removed storagePeer function
next step will be parallelization of processes

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4600 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-03-24 22:51:26 +00:00
orbiter
d3b06913ec protection against seed-db failure during enumeration 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4598 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-03-23 23:47:41 +00:00
orbiter
5aa96dbc36 fix for shutdown configuration 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4596 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-03-23 13:14:57 +00:00
orbiter
93633abed8 - removed some debugging code from search process - should speed up now 
- added some profiling code to search event - more time details in PerformanceSearch_p.html

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4594 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-03-23 00:55:04 +00:00
orbiter
fba46c51d7 fixed non-termination bug in qsort 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4593 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-03-22 23:15:28 +00:00
orbiter
541b817502 refactoring of switchboard queueing 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4591 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-03-22 01:28:37 +00:00
orbiter
fc94fbe224 another improvement to the collection sorting 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4589 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-03-20 23:11:04 +00:00
orbiter
11270d450e better quicksort-pivot computation: 30% faster (measured with test program) 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4588 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-03-20 22:01:12 +00:00
orbiter
3e44293f07 - fixed a problem with thread pools in row collection 
- added a line-viewing feature in threaddump	

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4587 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-03-20 14:21:58 +00:00
danielr
e43051b125 - fixed Threaddump output (html-escaped ie. <init>) 
- in EcoFS converted comments to javadoc


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4586 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-03-20 10:20:55 +00:00
orbiter
433ff855f7 - fixed another concurrency problem in collection sorting 
- fixed a typing problem that was introduced in svn 4579 and caused the crawl monitor to fail

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4585 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-03-19 23:47:24 +00:00
orbiter
19286fa2d1 tried to fix seed2.old.db-problem 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4584 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-03-19 22:35:19 +00:00
apfelmaennchen
5fde618337 changed display of y-marks 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4583 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-03-19 19:54:35 +00:00
orbiter
f3996e63b8 tried to fix more deadlocks: 
- changed connection modes in ftpc
- replaced sort tread pool in row collections by new one using util.concurrent. the old pool had caused blockings

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4582 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-03-19 11:23:43 +00:00
daburna
26155cbb5e update 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4581 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-03-18 17:53:54 +00:00