Marown/yacy_search_server
1
0
Fork 0
mirror of https://github.com/yacy/yacy_search_server.git synced 2024-09-19 00:01:41 +02:00
Code Issues Actions Packages Projects Releases Wiki Activity
6,523 Commits 1 Branch 20 Tags 836 MiB
a7d038bb7a
Commit Graph

192 Commits

Author SHA1 Message Date
orbiter
dd459281c8 applied code changes that are recommended by PMD 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6563 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2010-01-10 23:09:48 +00:00
orbiter
4a5100789f replaced _all_ size() == 0 with isEmpty() and all size() > 0 with !isEmpty(). The isEmpty() method is much faster in some cases, especially when used to access badly balanced hashtables where an size() operation becomes a large iteration. 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6510 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-12-02 00:37:59 +00:00
orbiter
4431b9767e added about 450 replacements for printStackTrace() methods to pipe such traces into the log at DATA/LOG/ 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6458 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-11-05 20:28:37 +00:00
orbiter
30f108f97d added stub of oai-pmh importer (not working yet) 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6437 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-10-30 15:58:04 +00:00
orbiter
3528b970d6 - refactoring 
- added new experimental (not-yet-working) image parser
- added new test image

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6431 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-10-19 22:34:44 +00:00
orbiter
b79f4f062f refactoring of yacy documents and parsers: they depend now only on the kelondro classes 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6426 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-10-18 00:53:43 +00:00
orbiter
e7f18ba24b refactoring 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6399 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-10-11 00:24:42 +00:00
orbiter
ce8dc575ca refactoring 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6398 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-10-11 00:12:19 +00:00
orbiter
bea3b99aff moved table and util classes 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6397 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-10-10 01:14:19 +00:00
orbiter
4446acc8cd moved kelondro order 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6392 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-10-09 23:22:22 +00:00
orbiter
f677d534b1 start of a really extensive refactoring which will produce a hierarchical package structure with the domain yacy.net as package root 
- moved here the logging classes as part of the new net.yacy.kelondro package

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6391 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-10-09 23:13:30 +00:00
lotus
386b9f35f6 activated resource observer for windows 7 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6378 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-10-07 06:20:24 +00:00
orbiter
04a548a1e3 - temporary integrated the transferURL servlet as static class instead as a class that is called using reflection to investigate the OOM problems in that class 
- fixes for numerous other problems
- removed dead code
- resdesign of the strings-method, which produces now less memory overhead and may help to prevent OOMs
- another fix for the deadlock problem in SplitTable

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6373 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-10-05 20:11:41 +00:00
orbiter
68465c37af added a convenience class to add files into a YaCy index 
to make this possible, the yacyURL must be able to process file:// urls, which has also been implemented
testing of the new class resulted in some bugfixes in other classes

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6313 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-09-14 21:17:42 +00:00
orbiter
2740d9dd79 added integration of osm maps for search 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6291 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-09-04 14:32:36 +00:00
orbiter
1762a7bcd6 - moved DidYouMean to the data package 
- added a DidYouMeanLibrary class that shall support the did you mean function with additional word lists


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6281 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-09-01 13:04:35 +00:00
orbiter
1d8d51075c refactoring: 
- removed the plasma package. The name of that package came from a very early pre-version of YaCy, even before YaCy was named AnomicHTTPProxy. The Proxy project introduced search for cache contents using class files that had been developed during the plasma project. Information from 2002 about plasma can be found here:
http://web.archive.org/web/20020802110827/http://anomic.de/AnomicPlasma/index.html
We stil have one class that comes mostly unchanged from the plasma project, the Condenser class. But this is now part of the document package and all other classes in the plasma package can be assigned to other packages.
- cleaned up the http package: better structure of that class and clean isolation of server and client classes. The old HTCache becomes part of the client sub-package of http.
- because the plasmaSwitchboard is now part of the search package all servlets had to be touched to declare a different package source.

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6232 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-07-19 20:37:44 +00:00
orbiter
ca72ed7526 -removed superfluous crawl cache 
-refactoring of crawler classes

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6221 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-07-15 21:07:46 +00:00
orbiter
49bbb9bd45 replaced tar library with integrated apache ant tar lib 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6212 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-07-14 11:31:40 +00:00
f1ori
f814e0fa81 enable warnings and fix most of it 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6196 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-07-11 21:01:27 +00:00
orbiter
57a88d435b redesign of parser mime type detection and parser steering 
There is now a mime-blacklist instead of a mime-whitelist

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6190 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-07-10 14:22:17 +00:00
orbiter
21b8704fb4 refactoring of the ParserDispatcher and ParserConfig: resulted into Idiom, Parser and Classification classes 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6188 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-07-09 22:25:31 +00:00
orbiter
dafffd0153 refactoring of parsers and document processing 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6182 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-07-08 21:48:08 +00:00
orbiter
77d2a3782c removed strange debugging strings 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6177 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-07-06 15:21:43 +00:00
lotus
7f868ca3c2 resource observer: support for yacyroot\DATA on an NTFS hardlink (Windows) 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6162 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-07-02 10:02:20 +00:00
orbiter
1f1399e5c5 extending visibility of objects and methods to avoid synthetic accessor methods and increase performance 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6156 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-06-30 13:25:46 +00:00
orbiter
154bbc3364 code cleanup: call of static methods directly to the class 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6155 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-06-30 13:01:35 +00:00
orbiter
222850414e simplification of the code: removed unused classes, methods and variables 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6154 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-06-30 09:27:46 +00:00
apfelmaennchen
a10c8022d1 DidYouMean: 
- limit the number of consumer threads to available CPUs
- added some javadoc

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6144 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-06-27 07:23:34 +00:00
orbiter
fd31a3616a - more logging in server process 
- fix for bas ascii in comment

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6084 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-06-16 15:10:59 +00:00
orbiter
ce1adf9955 serialized all logging using concurrency: 
high-performance search query situations as seen in yacy-metager integration showed deadlock situation caused by synchronization effects inside of sun.java code. It appears that the logger is not completely safe against deadlock situations in concurrent calls of the logger. One possible solution would be a outside-synchronization with 'synchronized' statements, but that would further apply blocking on all high-efficient methods that call the logger. It is much better to do a non-blocking hand-over of logging lines and work off log entries with a concurrent log writer. This also disconnects IO operations from logging, which can also cause IO operation when a log is written to a file. This commit not only moves the logger from kelondro to yacy.logging, it also inserts the concurrency methods to realize non-blocking logging.

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6078 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-06-15 21:19:54 +00:00
apfelmaennchen
39779e4796 DidYouMean: as I moved to only 8 consumer and 4 producer threads, I removed poison pills as it does not make sense anymore - threads are interrupted directly. Having a consumer thread per test case just didn't make sense either (see svn 6070) due to the massive overhead. 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6072 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-06-14 16:31:31 +00:00
apfelmaennchen
c3c4dd0933 DidYouMean - changed to much simpler LinkedBlockingQueue 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6071 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-06-14 15:25:57 +00:00
apfelmaennchen
01ac1b5d7e - blocking queue implementation of DidYouMean 
- timeout ist set to 500ms

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6070 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-06-14 11:53:09 +00:00
orbiter
b8bb1bb364 join with a timeout does not cause that the corresponding thread is stopped after the time-out. It does only cause that the waiting is stopped. Here we need additionally a signal to the thread to stop after we finished waiting. 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6069 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-06-13 23:54:52 +00:00
orbiter
b69f22e9ca mistake in last commit: computation of loops in ReversingTwoConsecutiveLetters 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6068 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-06-13 23:37:51 +00:00
orbiter
3130334932 - start first with threads that run more loops 
- join first with threads that run less loops

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6067 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-06-13 23:34:16 +00:00
apfelmaennchen
6cde7ebf16 DidYouMean 
- without I/O intensive sorting by count
- but with multiple threads

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6066 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-06-13 23:16:14 +00:00
orbiter
7c4d1d471c hand-over of more specific object 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6062 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-06-13 10:22:25 +00:00
apfelmaennchen
09acfa66d1 - improved "did you mean" 
- added &meanCount= to query string
- &meanCount=0 ==> no suggestion, no performance loss
- sorting suggestions by sb.indexSegment.termIndex().count()

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6059 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-06-13 06:20:05 +00:00
apfelmaennchen
da6ce37f7b - fixed encoding problem 
- added limit to 10 suggestions

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6058 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-06-12 21:36:26 +00:00
apfelmaennchen
54a48b4184 - added "did you mean" to search page 
- currently works for single word queries only!

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6057 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-06-12 20:36:03 +00:00
orbiter
bead0006da replaced tmp file extensions by prt 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6033 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-06-06 18:09:58 +00:00
orbiter
89aeb318d3 enhanced the wikimedia dump import process 
enhanced the wiki parser and condenser speed

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5931 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-05-08 10:36:13 +00:00
orbiter
5fb77116c6 added a submenu to index administration to import a wikimedia dump (i.e. a dump from wikipedia) into the YaCy index: see 
http://localhost:8080/IndexImportWikimedia_p.html

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5930 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-05-08 07:54:10 +00:00
orbiter
c097531e3d added a catch Exception to all thread to check if any of them silently dies without any other notification 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5922 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-05-05 06:31:35 +00:00
orbiter
9c6ac43f66 fixes for wiki parser 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5905 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-04-30 22:03:35 +00:00
orbiter
d079d6dfdb small changes in surrogate reader, wiki code and portal test 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5894 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-04-27 20:30:43 +00:00
orbiter
2e3186189b fix for mediawikiIndex surrogate producer + added concurrency 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5880 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-04-25 21:52:21 +00:00
orbiter
1b9e532c87 some concurrency for wikipedia dump reader 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5855 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-04-22 17:43:27 +00:00