Commit Graph

1198 Commits

Author SHA1 Message Date
orbiter
0e88ba997e * added option to generate url-lists as plain text file or in html
* modified generation of dom-lists so that they can be also generated as html
these options can be called as:
java -classpath classes yacy -domlist -format html
java -classpath classes yacy -domlist -format html .
java -classpath classes yacy -domlist -format text .
java -classpath classes yacy -urllist -format html .
java -classpath classes yacy -urllist -format text .
the -format <type> can be ommitted. The text is default
a home path can be asserted or omitted at the end of the parameters

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1178 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-12-07 01:40:52 +00:00
orbiter
13fdebc50d added authentication for link deletion in search result
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1177 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-12-07 00:36:05 +00:00
orbiter
37f88b4017 code cleanup
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1176 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-12-06 23:51:29 +00:00
orbiter
ec2b39c1ce code cleanup
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1175 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-12-06 22:30:15 +00:00
daburna
e078a50e0b - corrected spelling in IndexCreateWWWLocalQueue_p.html and simple_search.html
- updated de.lng with translation for simple_search.html and update translation for IndexCreateWWWLocalQueue_p.html

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1174 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-12-06 21:31:52 +00:00
orbiter
76618442e0 code cleanup
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1173 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-12-06 21:21:14 +00:00
orbiter
8f1f2daa5e implemented interactive link deletion of search results.
next steps: attach voting and restrict to administrator
to see the deletion button, move the mouse pointer to the left of a search result

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1172 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-12-06 16:15:21 +00:00
theli
6d0f7e6988 *) Adding missing file
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1171 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-12-06 11:20:06 +00:00
theli
44fa94ac52 *) Modifications for dbImport functionality
- dbImporter threads are now shutdown by the switchboard on server shutdown
   - adding possibility to pause a importer thread via GUI
   - Bugfix for abort function
     See: http://www.yacy-forum.de/viewtopic.php?p=13363#13363

*) Modification of content parser configuration
   - now it's possible to configure which parsers should be enabled for the proxy,
     crawler, icap, etc. separately
   - 

*) htmlFilterContentScraper.java
   - adding regular expression to normalize URLs containing /../ and /./ parts

*) httpc.java
   - adding functionality to unzip gzipped content
   - requested by roland: should be used later to allow gzipped seed lists

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1170 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-12-06 10:41:19 +00:00
orbiter
28ddba8813 added vote-button images
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1169 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-12-06 00:16:41 +00:00
low012
d5c36c8e2e *) now showing the total number of entries in the queue in addition to the number of entries in the list
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1168 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-12-05 23:24:46 +00:00
orbiter
dc778659fb fixed problem with time-out during result joint which caused OR behavior instead of AND beahvior
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1167 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-12-05 15:48:45 +00:00
orbiter
3d8a5ae652 code cleanup
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1166 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-12-05 14:24:13 +00:00
allo
ae6a4650bc reenabling debugMode (60 Seconds timeout for *all* http connections)
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1165 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-12-05 11:14:11 +00:00
theli
bb1f73ec15 *) Bugfix for code cleanup
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1164 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-12-05 09:48:11 +00:00
orbiter
7920e1547d code cleanup
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1163 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-12-05 09:13:13 +00:00
theli
972b8ddd2a *) minor changes: using constants properly
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1162 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-12-05 09:18:47 +00:00
theli
1c3750de57 *) Bugfix for code cleanup
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1161 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-12-05 09:15:05 +00:00
theli
64478b1f02 *) Adding possibility to delete crawler queue entries using regular expressions
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1160 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-12-05 09:11:28 +00:00
orbiter
1d6a6d1f85 code cleanup
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1159 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-12-05 00:17:12 +00:00
orbiter
a04930f025 code cleanup
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1158 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-12-04 23:51:28 +00:00
orbiter
548f0c6aff first Try with Eclipse / cleaned sources
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1157 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-12-04 23:14:30 +00:00
low012
d35fb38c2b *) making the serach button work with Opera, Konqueror and Safari again. Roland's code is still in there, but in a comment to be reviewed later
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1156 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-12-03 10:18:28 +00:00
low012
90b0eb144e just a typo...
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1155 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-12-03 09:58:00 +00:00
theli
129b15f3e1 *) Correcting logging output of db importer thread
See: http://www.yacy-forum.de/viewtopic.php?t=1555

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1154 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-12-02 11:56:12 +00:00
theli
7028412c54 *) Bugfix for NullpointerException in Java Finalizer
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1153 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-12-02 10:57:31 +00:00
orbiter
420d56ce79 extended db-testing
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1152 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-12-02 01:51:37 +00:00
orbiter
ecf765ec33 temporary fix to make jrpm extension compilable with my netbeans environment
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1151 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-12-01 23:03:54 +00:00
rramthun
8c0d982191 1.) Fix from Martin (he ist not at home...)
2.) Search button now gets blocked if clicked and JS active.

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1150 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-12-01 18:26:39 +00:00
theli
b604654c25 *) Adding possibility to do a settings migration on yacy startup
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1149 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-12-01 13:39:38 +00:00
theli
eb9daccfd2 *) Corrections in jMimeMagic MagicRule-file to detect rpm files properly
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1148 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-12-01 12:13:50 +00:00
theli
8ed0aaae8d *) Adding content Parser for RPM Files
- at the moment only the metadata is extracted

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1147 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-12-01 10:47:00 +00:00
theli
0f769215b5 *) urlRedirector now uses http head requests to determine the mimetype
of a resource before it checks if a URL has to be rejected or not.
   

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1146 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-11-30 04:52:22 +00:00
theli
b62c36d2ab *) Bugfix for urlRedirector
- media extension was not parsed correctly

*) Displaying urlRedirector connections in Connections_p.html

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1145 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-11-29 23:45:17 +00:00
allo
8f4259f1e3 md5Auth
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1144 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-11-29 21:31:06 +00:00
theli
818d37ce44 *) Removing getSimpleName
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1143 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-11-29 12:50:13 +00:00
theli
25e5593d53 *) Removing compile statement for www directory
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1142 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-11-29 12:47:19 +00:00
theli
b35c5a48bf *) First version of urlRedirector.pl script
- with this script it's possible to pass URLs from squid
     to yacy via the squid redirector interface
   - this URLs are then used by YaCy to feed the crawler

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1141 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-11-29 12:27:03 +00:00
theli
bdf30117c1 *) Redesign of parser configuration
- restructuring of mimeTypes based on the parsers
   - displaying parser usage count
   - displaying human readably parser names
   - displaying parser version information

*) httpdFileHandler.java
   - adding possibility to support "streaming" servlets
     which are special servlets that can communicate with
     the client via the connection streams autonomous
   - the name of these new servlet types must end with the 
     file extension .stream
   - this feature will be needed by the yacy ScreenSaver
     class to fetch statistic data from the peer without the
     need to reconnect to the server all the time

*) Adding human readable names and version information for
   all supported parsers

*) plasmaParser.java
   - adding new structure to store parser statistic data

*) Adding openDocument parser
   - can be used to parse odt files

*) jmimemagic
   - adding rules to detect openDocument formats properly

*) serverLog.java
   - adding functions that can be used to query if a given
     logging level is enabled or not.


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1140 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-11-29 07:27:58 +00:00
theli
3037c1e73b *) Removing compile statement for www directory
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1139 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-11-29 07:14:55 +00:00
allo
b86d1085e2 passwordAuth
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1138 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-11-28 22:35:41 +00:00
low012
edaa820bec resubmitting Allos patch after accidentally removing it
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1137 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-11-28 21:42:01 +00:00
low012
c45e47bf91 fixed vulnerability, see http://www.yacy-forum.de/viewtopic.php?t=1535
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1136 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-11-28 16:51:59 +00:00
theli
5bf70e6e14 *) Bugfix for serverClassLoader.java
- Classloading didn't work properly if there are multiple classes with the same name
   - This could occure because the yacy servlets have no package name defined and therefore
     are all in the same (default) package.

*) Bugfix for Duplicated Class Error
   See: http://www.yacy-forum.de/viewtopic.php?t=1341

  

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1135 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-11-28 10:15:25 +00:00
theli
d4ac3e25b1 *) Bugfix for file system link bug during detection of invalid URLs
See: http://www.yacy-forum.de/viewtopic.php?p=13301

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1134 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-11-28 07:17:43 +00:00
orbiter
adf75bc9fa better logging for invalid file path detection
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1133 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-11-27 22:55:30 +00:00
orbiter
40621a5663 anhancements in ranking preparation and fixed problem with parser/mime recognition
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1132 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-11-27 11:55:24 +00:00
allo
38d915e24c limiting the Number of Items displayed to 100 max.
http://www.yacy-forum.de/viewtopic.php?p=13275


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1131 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-11-27 09:50:23 +00:00
theli
c650b112ea *) Bugfix for relative URL Bug in Crawler
See: http://www.yacy-forum.de/viewtopic.php?p=13266#13266

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1130 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-11-27 06:35:23 +00:00
theli
7e670894d9 *) Suppressing stackTraces in proxyError message for "connect timed out" errors
See: http://www.yacy-forum.de/viewtopic.php?t=1504
*) Increasing default http client timeout

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1129 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-11-25 00:40:35 +00:00