Marown/yacy_search_server
1
0
Fork 0
mirror of https://github.com/yacy/yacy_search_server.git synced 2024-09-19 00:01:41 +02:00
Code Issues Actions Packages Projects Releases Wiki Activity
5,889 Commits 1 Branch 20 Tags 836 MiB
13c63f4082
Commit Graph

5889 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
orbiter
13c63f4082 a set of small fixes to crawling behaviour 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6216 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-07-15 14:15:51 +00:00
orbiter
a564df3984 update to mime types in parsers and httpd.mime 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6215 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-07-15 14:10:29 +00:00
orbiter
43c8defd79 enhanced parser with more extension + mime attributes 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6214 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-07-14 13:32:53 +00:00
orbiter
aee35bff6f replaced StringBuffer with StringBuilder in tar lib 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6213 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-07-14 13:31:57 +00:00
orbiter
49bbb9bd45 replaced tar library with integrated apache ant tar lib 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6212 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-07-14 11:31:40 +00:00
orbiter
f987fc6b4a added tar classes from apache ant tools 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6211 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-07-14 11:25:40 +00:00
orbiter
f2d4b6d7fa added tar classes from apache ant tools 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6210 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-07-14 11:25:05 +00:00
orbiter
b2263bc720 enhanced document type recognition 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6209 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-07-14 11:01:05 +00:00
lotus
aa38eb5a20 * maxfilesize -1 for infinite filesize 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6208 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-07-14 08:39:39 +00:00
orbiter
7d493cf8cc moved parser configuration in separate servelet 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6207 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-07-14 06:57:13 +00:00
lotus
9cfe89c8fc * process content-length as soon as it is received 
* corrected indentation

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6206 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-07-13 19:55:13 +00:00
orbiter
5240d22773 removed unused library jsmooth 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6205 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-07-13 18:16:03 +00:00
orbiter
3d26161dd1 removed unused libraries 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6204 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-07-13 14:47:09 +00:00
orbiter
50cf80056f removed jmimemagic library 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6203 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-07-13 10:58:37 +00:00
orbiter
e3c7f61145 removed unused libraries 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6202 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-07-13 10:21:22 +00:00
orbiter
3f113f38a8 removed unused imports 
removed unused libs from eclipse class path

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6201 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-07-13 10:19:10 +00:00
lotus
9f083bb6b2 check filetype before loading (no more mp4 loading) 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6200 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-07-12 16:50:11 +00:00
low012
b118bdd994 *) Deleted obsolete license file. 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6199 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-07-12 16:38:13 +00:00
f1ori
076ae02c44 * added pl and py to extensions excepted by htmlParser 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6198 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-07-12 16:35:35 +00:00
f1ori
d5e51cfd09 * workaround for non-working build property replacements 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6197 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-07-12 09:38:03 +00:00
f1ori
f814e0fa81 enable warnings and fix most of it 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6196 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-07-11 21:01:27 +00:00
f1ori
8931c8d6b4 improvments to debianpackage: 
* autoupdate completely disabled, display hint
* restart-button in interface works!

* moved all build-Variables to yacyBuildProperties
* fixed some warnings


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6195 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-07-11 17:03:22 +00:00
low012
fc1dc38b55 *) added spaces to make sure that no words are concatinated by accident 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6194 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-07-10 22:51:53 +00:00
low012
f242e7d7bc *) using Apache POI library to parse Word documents now 
*) removed tm-extractors library (can be found at http://www.textmining.org/ if necessary again)

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6193 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-07-10 22:49:16 +00:00
orbiter
caedd72400 - enhanced logging and exception details for parsers 
- removed inconsistencies in mime type declaration (one mime type should only appear once in all parsers)

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6192 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-07-10 16:15:33 +00:00
orbiter
4b74ad0a46 fixed setting of parser configuration servlets 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6191 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-07-10 15:02:34 +00:00
orbiter
57a88d435b redesign of parser mime type detection and parser steering 
There is now a mime-blacklist instead of a mime-whitelist

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6190 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-07-10 14:22:17 +00:00
lotus
e15d27bc63 avoiding double/wrong parser errors 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6189 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-07-10 09:31:10 +00:00
orbiter
21b8704fb4 refactoring of the ParserDispatcher and ParserConfig: resulted into Idiom, Parser and Classification classes 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6188 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-07-09 22:25:31 +00:00
orbiter
8ca1f5d400 - some work to integrate the html parser the same way as the other parsers are integrated (not finished) 
- added migration of code of settings pages (hmm.. does not work correctly yet, sorry)
- more refactoring
- removed more unused code

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6187 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-07-09 20:56:30 +00:00
low012
1ee109761f *) added changes which were lost 
*) additional annotations
*) additional svn properties
*) _no_ functional changes

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6186 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-07-09 20:13:11 +00:00
orbiter
499723891d removed all non-http daemons; they had not been used and may be a potential security risk. 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6185 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-07-08 22:24:34 +00:00
orbiter
0e8647d62f refactoring of search classes 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6184 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-07-08 22:14:57 +00:00
orbiter
92be8f39bf removed unused packages 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6183 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-07-08 22:08:35 +00:00
orbiter
dafffd0153 refactoring of parsers and document processing 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6182 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-07-08 21:48:08 +00:00
low012
8041e91f56 *) Ooops! 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6181 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-07-08 20:58:47 +00:00
low012
69551ff3d9 *) added several MIME types (derived from http://filext.com/), some of them might be rather uncommon 
*) added an annotation forgotten in last commit

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6180 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-07-08 20:53:08 +00:00
low012
11dfb2d54f minor changes: 
*) added annotations
*) set svn properties and added keywords to comments of parser classes
*) made a variable final to prevent (theoretical case of) change of object instance in synchronized block

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6179 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-07-08 19:20:03 +00:00
apfelmaennchen
ba7905aaa8 PortalSearch: 
- removed buttons
- auto load on scroll
- some small fixes

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6178 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-07-08 19:01:06 +00:00
orbiter
77d2a3782c removed strange debugging strings 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6177 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-07-06 15:21:43 +00:00
lotus
4320f69574 universal handling for crashed parsers 
reverting r6090/1

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6176 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-07-05 08:12:14 +00:00
lotus
535aee4425 update for windows build 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6175 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-07-05 08:09:50 +00:00
orbiter
801aa08162 added f1oris update location 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6174 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-07-04 20:56:25 +00:00
orbiter
024744245c small refactoring to prepare for new queues 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6173 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-07-04 12:17:10 +00:00
orbiter
16efcd0366 fix for http://forum.yacy-websuche.de/viewtopic.php?f=6&t=2252&hilit=&p=16389#p16389 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6172 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-07-04 06:24:58 +00:00
f1ori
0f3246e90a * fix debian package 
* add Class containing buildvariables


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6171 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-07-03 21:57:05 +00:00
f1ori
8544cfd5a6 * remove seperate build-files for parsers 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6170 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-07-03 20:06:06 +00:00
orbiter
24cb6d68bc - renamed Stack to RecordStack to avoid name confusion with new classes 
- added new Stack class that implements a stack on BLOB files
- added new Stacks class that can be used for a set of Stacks (a 'Stack Database')
- added methods to other classes to support the new stacks

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6169 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-07-03 16:35:34 +00:00
lotus
ec2970cc76 higher dht distribution speed by default 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6168 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-07-03 13:28:19 +00:00
orbiter
995da28c73 all stack/heap files that had been stored in DATA/PLASMA are now stored in the network-specific QUEUES path 
There is no migration. All crawls must be restarted.

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6167 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-07-02 17:01:23 +00:00