Commit Graph

808 Commits

Author SHA1 Message Date
theli
8862b6ba4b *) Corrections for code cleanup 1175
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1179 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-12-07 08:15:39 +00:00
orbiter
0e88ba997e * added option to generate url-lists as plain text file or in html
* modified generation of dom-lists so that they can be also generated as html
these options can be called as:
java -classpath classes yacy -domlist -format html
java -classpath classes yacy -domlist -format html .
java -classpath classes yacy -domlist -format text .
java -classpath classes yacy -urllist -format html .
java -classpath classes yacy -urllist -format text .
the -format <type> can be ommitted. The text is default
a home path can be asserted or omitted at the end of the parameters

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1178 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-12-07 01:40:52 +00:00
orbiter
13fdebc50d added authentication for link deletion in search result
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1177 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-12-07 00:36:05 +00:00
orbiter
37f88b4017 code cleanup
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1176 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-12-06 23:51:29 +00:00
orbiter
ec2b39c1ce code cleanup
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1175 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-12-06 22:30:15 +00:00
orbiter
76618442e0 code cleanup
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1173 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-12-06 21:21:14 +00:00
orbiter
8f1f2daa5e implemented interactive link deletion of search results.
next steps: attach voting and restrict to administrator
to see the deletion button, move the mouse pointer to the left of a search result

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1172 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-12-06 16:15:21 +00:00
theli
6d0f7e6988 *) Adding missing file
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1171 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-12-06 11:20:06 +00:00
theli
44fa94ac52 *) Modifications for dbImport functionality
- dbImporter threads are now shutdown by the switchboard on server shutdown
   - adding possibility to pause a importer thread via GUI
   - Bugfix for abort function
     See: http://www.yacy-forum.de/viewtopic.php?p=13363#13363

*) Modification of content parser configuration
   - now it's possible to configure which parsers should be enabled for the proxy,
     crawler, icap, etc. separately
   - 

*) htmlFilterContentScraper.java
   - adding regular expression to normalize URLs containing /../ and /./ parts

*) httpc.java
   - adding functionality to unzip gzipped content
   - requested by roland: should be used later to allow gzipped seed lists

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1170 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-12-06 10:41:19 +00:00
orbiter
28ddba8813 added vote-button images
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1169 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-12-06 00:16:41 +00:00
orbiter
dc778659fb fixed problem with time-out during result joint which caused OR behavior instead of AND beahvior
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1167 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-12-05 15:48:45 +00:00
orbiter
3d8a5ae652 code cleanup
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1166 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-12-05 14:24:13 +00:00
allo
ae6a4650bc reenabling debugMode (60 Seconds timeout for *all* http connections)
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1165 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-12-05 11:14:11 +00:00
theli
bb1f73ec15 *) Bugfix for code cleanup
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1164 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-12-05 09:48:11 +00:00
orbiter
7920e1547d code cleanup
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1163 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-12-05 09:13:13 +00:00
theli
972b8ddd2a *) minor changes: using constants properly
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1162 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-12-05 09:18:47 +00:00
theli
1c3750de57 *) Bugfix for code cleanup
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1161 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-12-05 09:15:05 +00:00
theli
64478b1f02 *) Adding possibility to delete crawler queue entries using regular expressions
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1160 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-12-05 09:11:28 +00:00
orbiter
1d6a6d1f85 code cleanup
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1159 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-12-05 00:17:12 +00:00
orbiter
a04930f025 code cleanup
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1158 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-12-04 23:51:28 +00:00
low012
90b0eb144e just a typo...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1155 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-12-03 09:58:00 +00:00
theli
129b15f3e1 *) Correcting logging output of db importer thread
See: http://www.yacy-forum.de/viewtopic.php?t=1555

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1154 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-12-02 11:56:12 +00:00
theli
7028412c54 *) Bugfix for NullpointerException in Java Finalizer
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1153 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-12-02 10:57:31 +00:00
orbiter
420d56ce79 extended db-testing
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1152 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-12-02 01:51:37 +00:00
orbiter
ecf765ec33 temporary fix to make jrpm extension compilable with my netbeans environment
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1151 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-12-01 23:03:54 +00:00
rramthun
8c0d982191 1.) Fix from Martin (he ist not at home...)
2.) Search button now gets blocked if clicked and JS active.

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1150 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-12-01 18:26:39 +00:00
theli
b604654c25 *) Adding possibility to do a settings migration on yacy startup
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1149 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-12-01 13:39:38 +00:00
theli
8ed0aaae8d *) Adding content Parser for RPM Files
- at the moment only the metadata is extracted

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1147 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-12-01 10:47:00 +00:00
theli
0f769215b5 *) urlRedirector now uses http head requests to determine the mimetype
of a resource before it checks if a URL has to be rejected or not.
   

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1146 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-11-30 04:52:22 +00:00
theli
b62c36d2ab *) Bugfix for urlRedirector
- media extension was not parsed correctly

*) Displaying urlRedirector connections in Connections_p.html

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1145 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-11-29 23:45:17 +00:00
allo
8f4259f1e3 md5Auth
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1144 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-11-29 21:31:06 +00:00
theli
818d37ce44 *) Removing getSimpleName
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1143 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-11-29 12:50:13 +00:00
theli
b35c5a48bf *) First version of urlRedirector.pl script
- with this script it's possible to pass URLs from squid
     to yacy via the squid redirector interface
   - this URLs are then used by YaCy to feed the crawler

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1141 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-11-29 12:27:03 +00:00
theli
bdf30117c1 *) Redesign of parser configuration
- restructuring of mimeTypes based on the parsers
   - displaying parser usage count
   - displaying human readably parser names
   - displaying parser version information

*) httpdFileHandler.java
   - adding possibility to support "streaming" servlets
     which are special servlets that can communicate with
     the client via the connection streams autonomous
   - the name of these new servlet types must end with the 
     file extension .stream
   - this feature will be needed by the yacy ScreenSaver
     class to fetch statistic data from the peer without the
     need to reconnect to the server all the time

*) Adding human readable names and version information for
   all supported parsers

*) plasmaParser.java
   - adding new structure to store parser statistic data

*) Adding openDocument parser
   - can be used to parse odt files

*) jmimemagic
   - adding rules to detect openDocument formats properly

*) serverLog.java
   - adding functions that can be used to query if a given
     logging level is enabled or not.


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1140 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-11-29 07:27:58 +00:00
allo
b86d1085e2 passwordAuth
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1138 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-11-28 22:35:41 +00:00
theli
5bf70e6e14 *) Bugfix for serverClassLoader.java
- Classloading didn't work properly if there are multiple classes with the same name
   - This could occure because the yacy servlets have no package name defined and therefore
     are all in the same (default) package.

*) Bugfix for Duplicated Class Error
   See: http://www.yacy-forum.de/viewtopic.php?t=1341

  

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1135 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-11-28 10:15:25 +00:00
theli
d4ac3e25b1 *) Bugfix for file system link bug during detection of invalid URLs
See: http://www.yacy-forum.de/viewtopic.php?p=13301

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1134 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-11-28 07:17:43 +00:00
orbiter
adf75bc9fa better logging for invalid file path detection
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1133 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-11-27 22:55:30 +00:00
orbiter
40621a5663 anhancements in ranking preparation and fixed problem with parser/mime recognition
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1132 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-11-27 11:55:24 +00:00
theli
c650b112ea *) Bugfix for relative URL Bug in Crawler
See: http://www.yacy-forum.de/viewtopic.php?p=13266#13266

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1130 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-11-27 06:35:23 +00:00
theli
7e670894d9 *) Suppressing stackTraces in proxyError message for "connect timed out" errors
See: http://www.yacy-forum.de/viewtopic.php?t=1504
*) Increasing default http client timeout

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1129 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-11-25 00:40:35 +00:00
theli
4e73035aef *) Bugfix for "too many open files" during index distribution
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1128 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-11-24 21:47:16 +00:00
allo
d8afe60e07 Bugfix for last Bugfix ;-).
host/port were set to originaladdress instead of the correct values for the new Url.


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1126 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-11-23 14:05:25 +00:00
orbiter
1b656f6b31 correction of bug from svn 1123
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1125 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-11-23 12:07:07 +00:00
orbiter
f57e2d67f5 shortened network overview (less columns fit easier on page)
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1124 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-11-23 11:57:30 +00:00
allo
24d15eb0e8 moving the redirector code
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1123 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-11-23 07:52:36 +00:00
allo
787c368696 synchronized redirectors and using the port.
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1122 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-11-23 07:37:15 +00:00
orbiter
85282b1d98 enhanced YBR recognition and search result heuristics
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1121 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-11-23 01:40:02 +00:00
allo
4776f3f815 squid like redirctors
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1120 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-11-22 22:07:29 +00:00
orbiter
b9cc9029e3 added ybr selection for remote search
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1119 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-11-22 16:10:24 +00:00