Commit Graph

878 Commits

Author SHA1 Message Date
allo
8f4259f1e3 md5Auth
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1144 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-11-29 21:31:06 +00:00
theli
818d37ce44 *) Removing getSimpleName
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1143 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-11-29 12:50:13 +00:00
theli
b35c5a48bf *) First version of urlRedirector.pl script
- with this script it's possible to pass URLs from squid
     to yacy via the squid redirector interface
   - this URLs are then used by YaCy to feed the crawler

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1141 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-11-29 12:27:03 +00:00
theli
bdf30117c1 *) Redesign of parser configuration
- restructuring of mimeTypes based on the parsers
   - displaying parser usage count
   - displaying human readably parser names
   - displaying parser version information

*) httpdFileHandler.java
   - adding possibility to support "streaming" servlets
     which are special servlets that can communicate with
     the client via the connection streams autonomous
   - the name of these new servlet types must end with the 
     file extension .stream
   - this feature will be needed by the yacy ScreenSaver
     class to fetch statistic data from the peer without the
     need to reconnect to the server all the time

*) Adding human readable names and version information for
   all supported parsers

*) plasmaParser.java
   - adding new structure to store parser statistic data

*) Adding openDocument parser
   - can be used to parse odt files

*) jmimemagic
   - adding rules to detect openDocument formats properly

*) serverLog.java
   - adding functions that can be used to query if a given
     logging level is enabled or not.


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1140 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-11-29 07:27:58 +00:00
allo
b86d1085e2 passwordAuth
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1138 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-11-28 22:35:41 +00:00
theli
5bf70e6e14 *) Bugfix for serverClassLoader.java
- Classloading didn't work properly if there are multiple classes with the same name
   - This could occure because the yacy servlets have no package name defined and therefore
     are all in the same (default) package.

*) Bugfix for Duplicated Class Error
   See: http://www.yacy-forum.de/viewtopic.php?t=1341

  

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1135 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-11-28 10:15:25 +00:00
theli
d4ac3e25b1 *) Bugfix for file system link bug during detection of invalid URLs
See: http://www.yacy-forum.de/viewtopic.php?p=13301

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1134 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-11-28 07:17:43 +00:00
orbiter
adf75bc9fa better logging for invalid file path detection
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1133 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-11-27 22:55:30 +00:00
orbiter
40621a5663 anhancements in ranking preparation and fixed problem with parser/mime recognition
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1132 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-11-27 11:55:24 +00:00
theli
c650b112ea *) Bugfix for relative URL Bug in Crawler
See: http://www.yacy-forum.de/viewtopic.php?p=13266#13266

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1130 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-11-27 06:35:23 +00:00
theli
7e670894d9 *) Suppressing stackTraces in proxyError message for "connect timed out" errors
See: http://www.yacy-forum.de/viewtopic.php?t=1504
*) Increasing default http client timeout

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1129 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-11-25 00:40:35 +00:00
theli
4e73035aef *) Bugfix for "too many open files" during index distribution
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1128 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-11-24 21:47:16 +00:00
allo
d8afe60e07 Bugfix for last Bugfix ;-).
host/port were set to originaladdress instead of the correct values for the new Url.


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1126 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-11-23 14:05:25 +00:00
orbiter
1b656f6b31 correction of bug from svn 1123
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1125 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-11-23 12:07:07 +00:00
orbiter
f57e2d67f5 shortened network overview (less columns fit easier on page)
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1124 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-11-23 11:57:30 +00:00
allo
24d15eb0e8 moving the redirector code
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1123 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-11-23 07:52:36 +00:00
allo
787c368696 synchronized redirectors and using the port.
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1122 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-11-23 07:37:15 +00:00
orbiter
85282b1d98 enhanced YBR recognition and search result heuristics
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1121 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-11-23 01:40:02 +00:00
allo
4776f3f815 squid like redirctors
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1120 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-11-22 22:07:29 +00:00
orbiter
b9cc9029e3 added ybr selection for remote search
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1119 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-11-22 16:10:24 +00:00
orbiter
0e25020f51 added first generation and usage of YBR index-files. Enhanced overall ranking of search results.
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1118 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-11-22 15:17:05 +00:00
allo
52a0237bf2 using Filetemplates for #[metas]# and other static includes
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1116 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-11-21 08:33:54 +00:00
theli
90d6c6223b *) Adding color codes to network graphic legend
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1114 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-11-21 08:13:01 +00:00
orbiter
bfe51c7228 added generation of domain-list
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1112 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-11-21 01:30:30 +00:00
orbiter
0ec54d9c5f enhanced CR-file handling and added first RCI-evaluation tests
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1110 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-11-20 18:55:35 +00:00
theli
99fb26e499 *) Suppressing stackTraces in proxyError message for harmless errors
See: http://www.yacy-forum.de/viewtopic.php?t=1504

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1108 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-11-20 15:37:23 +00:00
theli
c2fe3a1670 *) Updating jMimeMagic Ruleset
- to detect some special formated html documents correctly
   - adding rule to detect vCards
*) plasmaParser now supports parsing of files that have a supported fileExtension
   but a unsupported mimeType because the webserver has set it incorrectly to text/plain
*) Adding vCard new Parser


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1107 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-11-20 14:39:58 +00:00
orbiter
88e3234393 fine-tuning of rci-generation
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1105 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-11-18 02:00:25 +00:00
orbiter
a12759c1bf first try to implement a rci-computation from cr-files
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1103 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-11-17 16:17:56 +00:00
orbiter
4a8e8f269e refactoring of cr-processing; new kelondro class to handle the attribute file format
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1100 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-11-17 12:08:04 +00:00
orbiter
24dc0e0760 implemented cr-file processing and further transmission steps
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1099 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-11-17 01:59:01 +00:00
low012
5cd1e9cef4 *) fixed some dirty code, idea analog to bit stuffing
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1098 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-11-16 23:58:13 +00:00
orbiter
022530df7e small bugfix in kelondroTree
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1097 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-11-16 22:36:36 +00:00
orbiter
9d9a87f445 limited htcache storage length
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1096 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-11-16 18:40:44 +00:00
theli
8e308cf50e *) Possibility to change the server port on-the-fly.
- Now it's possible to change the server port without the need to restart the whole server.
   

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1089 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-11-15 15:03:15 +00:00
theli
d0dfccdb77 *) Making CrawlStacker pool configurable via GUI and config file
See: http://www.yacy-forum.de/viewtopic.php?t=1448

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1087 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-11-15 12:46:22 +00:00
theli
3631cb1f6d *) deleting empty entities during index selection
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1086 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-11-15 12:23:46 +00:00
theli
ca26aab9b1 *) More debugging output for migrateWords
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1085 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-11-15 11:55:09 +00:00
theli
9b35ae9027 *) Correcting wrong % values on IndexTransfer_p page
See: http://www.yacy-forum.de/viewtopic.php?p=12646 

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1084 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-11-15 09:52:50 +00:00
theli
e6bf9d90a5 *) Fixing Problems with MalformedURLs during Word Selection
- removing (lurl.toString() == null) comparison because toString() is never null
   - adding (lurl.url() == null) condition because url() is null if we have selected a word entry with
     a malformed URL

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1083 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-11-15 09:07:00 +00:00
theli
86a9210264 *) indexing queue slots are now configurable via config file
See: http://www.yacy-forum.de/viewtopic.php?t=1480

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1081 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-11-15 08:25:46 +00:00
theli
3c11d7b81c *) Bugfix for minimizeUrlDB
- function didn't work correctly because of new url hash structure
   See: http://www.yacy-forum.de/viewtopic.php?p=12753#12753

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1080 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-11-15 07:35:04 +00:00
orbiter
9913049009 fixed outOfMemory bug caused by loops in kelondroTree during enumeration
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1079 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-11-15 01:20:05 +00:00
allo
f8f9d509d5 removed dead Code
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1078 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-11-14 12:48:14 +00:00
allo
5918d3985e removed Debug Statements
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1076 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-11-14 11:47:46 +00:00
theli
bbb936b9ea *) Bugfix for not human readable content of PDFs while viewing the URL Content via GUI
- This Bug also affects the snippet generation on non html/text documents
   See: http://www.yacy-forum.de/viewtopic.php?t=1472

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1075 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-11-14 10:25:43 +00:00
theli
445e3a620f *) Avoid rejecting of html content by the crawler when the file extension is not set properly
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1074 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-11-14 10:19:40 +00:00
orbiter
a3fd0069f5 fixed bug in kelondroTree node iteration (rotation did not work)
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1072 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-11-14 00:23:20 +00:00
theli
fd58d5f8e6 *) Adding possibility to specify the interface / IP-Address where YaCy should bind to.
- e.g. Port = 192.168.0.1:8080
          Port = #eth0:8080
          Port = 8080

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1071 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-11-13 17:03:52 +00:00
allo
889de6686c Migration in yacyVersion
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1070 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-11-13 15:12:48 +00:00
theli
444a5a9368 *) Bugfix for Entries with null url in GlobalQueue
See: http://www.yacy-forum.de/viewtopic.php?p=12675#12675

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1069 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-11-13 14:59:38 +00:00
allo
3bbb932fa2 Bugfix for nullpointerexception.
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1067 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-11-13 09:55:14 +00:00
allo
4976f97956 Logout Function on User.html
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1065 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-11-12 15:29:18 +00:00
allo
b74e990c74 better Logout function
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1064 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-11-12 12:09:45 +00:00
borg-0300
ebac51df52 restore defaultRemoteProfile
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1063 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-11-12 11:38:35 +00:00
allo
4daa4b00e9 logout function
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1062 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-11-11 22:41:36 +00:00
borg-0300
5778428455 move cutUrlText to nxTools,
max length from URLs(title) on searchpage now 120 chars


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1060 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-11-11 13:40:53 +00:00
borg-0300
9158845c3b bugfix for snippet text null bytes
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1059 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-11-11 13:27:36 +00:00
orbiter
6e81f2580d try to fix bug with storage of settings
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1058 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-11-11 08:41:13 +00:00
orbiter
f763923e0a added missing files for last commit
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1057 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-11-11 08:02:46 +00:00
theli
9649d08171 *) More tolerant robots parser
- converting tabs to spaces
   - cutting of '*' in the disallow section

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1056 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-11-11 07:49:54 +00:00
orbiter
79818a320f introduced citation-rank transmission protocol and activate transport for anonymisation
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1055 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-11-10 23:48:20 +00:00
theli
6f8d7d3bcd *) Adding first version of YaCy bookmarklet
- this can be used to easily crawl a webpage which is currently opened in the browser
   - to get the bookmarklet javascript simply call http://localhost:8000/QuickCrawlLink_p.html
     and drag and drop the link shown to your Browsers Toolbar/Link-Bar.

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1053 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-11-08 12:14:51 +00:00
theli
7e0647f692 *) Bugfix for userDB usage during authentication
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1052 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-11-08 10:17:12 +00:00
theli
93cadb47b9 *) More tolerant robots parser for robots-files which missing empty lines between rule blocks
See: http://www.yacy-forum.de/viewtopic.php?p=12471

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1048 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-11-08 07:41:25 +00:00
orbiter
02f8013013 auto-delete of corrupted word files during word-migration
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1047 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-11-07 14:57:37 +00:00
orbiter
d2731418bf added creation of global ranking files and changed url normal form usage
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1046 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-11-07 12:33:02 +00:00
theli
6f9f8ed8f8 *) Automatic Reset of Stack Crawler DB on startup errors
See: http://www.yacy-forum.de/viewtopic.php?t=1432

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1045 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-11-07 12:19:05 +00:00
theli
fb766413d1 *) Changes on httpc dns caching
- Bugfix: old dns cache did not handle case insensitive hostnames correctly. 
   - adding a possibility to set domain name patterns defining hostnames that should not be cached by the httpc dns cache
     e.g. borg-300.dyndns.org
     This can be done by setting the new httpc.nameCacheNoCachingPatterns property
   - using httpc.dnsResolve wherever possible within the sourcecode
     [httpd.java,plasmaCrawlStacker.java]

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1044 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-11-07 10:57:54 +00:00
orbiter
bc420c62f6 fixed htcache path generation (never change a running system)
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1041 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-11-07 01:31:11 +00:00
borg-0300
795f488222 new urlNormalform version
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1040 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-11-06 22:35:56 +00:00
orbiter
c86d801b0f removed dyndns domains from dns caching
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1039 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-11-06 22:12:08 +00:00
orbiter
6dc42a2392 detecting of loops in kelondroTree during last/first-Node search
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1038 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-11-06 21:06:55 +00:00
borg-0300
17d2830394 see: http://www.yacy-forum.de/viewtopic.php?t=1416
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1037 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-11-06 17:24:57 +00:00
theli
dd24f0252f *) Searchword highlighting for info page
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1036 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-11-06 06:27:17 +00:00
theli
f9fb284fb7 *) Better handling of robots.txt files with incorrect keywords
See: http://www.yacy-forum.de/viewtopic.php?p=12292#12292

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1035 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-11-06 06:01:08 +00:00
borg-0300
a1406f4617 urlNormalform: no logging
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1034 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-11-05 16:21:04 +00:00
borg-0300
72cde1d894 getCachePath: no logging
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1033 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-11-04 22:47:13 +00:00
borg-0300
1fbd72f9e0 rename "index.html" to "ndx"
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1032 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-11-04 22:39:33 +00:00
borg-0300
cd1107d85e added support for URLs with '?&'
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1030 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-11-04 17:25:15 +00:00
borg-0300
5fb2b017cb small change
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1029 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-11-04 16:37:56 +00:00
borg-0300
60e869f236 bugfix for last commit
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1028 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-11-04 15:52:12 +00:00
borg-0300
544e4ea90e small change
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1027 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-11-04 14:11:46 +00:00
borg-0300
00ab4d8723 cleaned, small change, Properties
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1026 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-11-04 13:41:51 +00:00
theli
b8ceb1ffde *) Adding better https support for crawler
- solving problems with unkown certificates by implementing a dummy trust Manager
   - adding https support to robots-parser 
   - Seed File can now be downloaded from https resources
   - adapting plasmaHTCache.java to support https URLs properly

*) URL Normalization
   - sub URLs are now normalized properly during indexing
   - pointing urlNormalForm function of plasmaParser to htmlFilterContentScraper function
   - normalizing URLs which were received by a crawlOrder request

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1024 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-11-03 15:28:37 +00:00
borg-0300
d2507c6081 rename setJunior()... to orJunior()...,
added javadoc, 
added getPeerType(), setIP(), setPort(String port)


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1023 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-11-03 14:16:16 +00:00
borg-0300
e3179a6394 added getOwnSeedFile()
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1022 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-11-03 14:07:58 +00:00
borg-0300
a803a509ae bugfix: port handling in HTCache
grogram flow, cleared up


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1021 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-11-03 12:39:24 +00:00
theli
3d0dfd4df4 *) Using StringBuffer instead of String concatenation
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1018 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-11-03 08:08:37 +00:00
low012
452db479cd *) bugfix: "21" was displayed as "21" in yacyWiki
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1017 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-11-02 23:28:54 +00:00
hydrox
cb69047b91 *)cleanup access static methods and fields
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1016 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-11-02 17:56:26 +00:00
hydrox
56b9f34411 *)removed unused imports
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1015 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-11-02 16:30:45 +00:00
orbiter
5f68b6886b introduced new url-hashes for better ranking computation
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1013 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-11-02 00:54:55 +00:00
orbiter
aadace1285 fixed network image in search performance monitor
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1012 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-11-01 00:49:13 +00:00
orbiter
bb369c98de fixed search result ordering by date
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1011 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-10-31 17:17:48 +00:00
hydrox
295aff52a3 *)added offline-browsing-support (onlineMode=0)
*)online-mode now can be changed in Status.html

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1010 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-10-31 12:25:40 +00:00
orbiter
4d1e56e4d9 fixed intermission-bug (removed 'break for intermission' of httpd-thread)
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1009 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-10-31 10:46:13 +00:00
orbiter
b058ecf0bc refactoring of image-generation; added experimental PNG encoder (not active now)
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1008 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-10-31 02:43:55 +00:00
orbiter
7a9fc653e9 fixed 100% cpu bug in wikiCode
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1007 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-10-30 14:27:09 +00:00
orbiter
1aa707d357 fix for http://www.yacy-forum.de/viewtopic.php?p=11866#11866
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1005 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-10-29 15:30:06 +00:00
orbiter
d42531e1b2 added auto-reset for NURL-DBs
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1004 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-10-29 13:53:32 +00:00
allo
5526d2ff73 Basic WatchCrawler.html, which loades the Queues using AJAX.
ajax.js: generic Ajax-Functions
xml/queues/indexing_p*: the Backend


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1002 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-10-29 09:47:30 +00:00
allo
92c49b406b adminAuth with userDB and adminAuthenticated (fix for statuspage)
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1001 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-10-28 18:06:36 +00:00
rramthun
27f180f24b Update of YaWoStat to 0.2.
Now does not try to make 400000! operations to load a 4MB textfile :-/

Program is not finished yet.

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1000 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-10-28 16:27:49 +00:00
theli
3b5d0eb053 *) Synchronizing robots.txt downloads to avoid parallel downloads of the same file by separate threads
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@998 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-10-28 09:10:48 +00:00
theli
723e056c48 *) Bugfix for ClassCastException during SessionPool.close
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@996 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-10-28 07:25:22 +00:00
theli
e58e85363d *) Bugfix for ConcurrentModificationException while operating on seed properties
*) Bugfix for YACY database inconsistency (no more elements available in db '...seed.new.db'), re-set of db.
   See: http://www.yacy-forum.de/viewtopic.php?p=11836#11836
        http://www.yacy-forum.de/viewtopic.php?p=11814#11814

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@995 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-10-28 07:15:00 +00:00
orbiter
2dc12c45cb small corrections to profiling generation
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@994 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-10-28 01:29:54 +00:00
orbiter
d656e2b433 added a memory-profile chart generation to database performance testing
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@993 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-10-28 01:22:11 +00:00
theli
9319445f38 *) Minor changes to db performance test
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@992 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-10-27 11:37:21 +00:00
theli
ec3af327f7 *) Bugfix for Proxy-Authentication against remote proxy
See: http://www.yacy-forum.de/viewtopic.php?p=11804#11804

*) Adding first version of db test for mysql
   NOTES:
   - db user + db + db table must be created before starting the test
   - db table must be empty. Entries can not be updated at the moment
   - db connection properties must be changed in the sourcecode at the moment
   TODOs:
   - accepting connection properties via command line
   - implementing update + remove + read operations
   - 'maybe' adding code to create db + table if it doesn't exists

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@991 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-10-27 11:28:37 +00:00
orbiter
5b0911d7ea added new performance menu for search sequence configuration and monitoring
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@990 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-10-27 00:42:08 +00:00
orbiter
4fa942511b de-serialized read and write access
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@989 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-10-26 23:05:20 +00:00
orbiter
09a0e898e0 created database test application
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@988 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-10-26 15:31:58 +00:00
orbiter
1ff0ced515 integration of an interface class for abstract access of kelondro indexed structures like kelondroTree and kelondroHashtable
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@987 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-10-26 12:37:22 +00:00
allo
ada06b0674 bugfix for Networkimage from Hydrox
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@986 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-10-26 10:08:37 +00:00
allo
e3f87f9dab Bugfix for not parsable Numbers (NumberFormatException when loading users)
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@985 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-10-25 12:30:07 +00:00
theli
d3ad712418 *) Bugfix for Seed file upload problem via ftpc
See: http://www.yacy-forum.de/viewtopic.php?p=11662#11662

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@984 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-10-25 10:45:14 +00:00
theli
9a2afe88d4 *) Deactivating unlimited timeout for persistent connections because this
could cause problems with clients which do not shutdown persistent 
   connections properly.
   - Setting timeout for idle persistent connections to 30 minutes

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@983 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-10-25 10:21:00 +00:00
orbiter
1aa4ba8b62 added post-search filtering of redundant urls (longer than existing cited)
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@982 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-10-25 09:06:00 +00:00
low012
8e7b214e41 *) 5 inverted commas are needed instead of only 4 to strongly emphasise a text (closer to Wikipedia's syntax now)
*) Tried to fix nested <pre> and [= issues, but did not succeed due to sleepyness. Will continue work when I'm awake and have some time.


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@981 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-10-24 22:37:10 +00:00
orbiter
8d827cdb30 tried to fix problems with order of network list by last-seen (which could also improve the network picture)
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@980 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-10-24 14:07:43 +00:00
low012
1e6cf9fd41 *) indented texts between escape symbols behave like indented preformatted texts now
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@979 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-10-24 13:21:57 +00:00
theli
446e7e8bef *) Bugfix for Seed-Upload - Permission denied problem
See: http://www.yacy-forum.de/viewtopic.php?p=11648#11648

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@978 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-10-24 08:54:46 +00:00
orbiter
097009d910 experimental visualization of DHT access during global search (temporary)
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@977 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-10-24 00:34:15 +00:00
orbiter
4dcbc26ef1 introduction of search profiles; very experimental
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@976 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-10-23 17:50:27 +00:00
theli
6c48c3ce39 *) Bugfix for ArithmeticException during IndexTransfer
See: http://www.yacy-forum.de/viewtopic.php?t=1362

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@974 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-10-23 16:07:44 +00:00
low012
fd1b08303e fixed a bug theli pointed out (indented <pre>-blocks)
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@973 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-10-23 14:04:21 +00:00
theli
525c8dcbd4 *) Adding Traffic Statistic for Crawler
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@972 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-10-23 12:35:48 +00:00
theli
9a5ab62928 *) Adding yacy specific X-YACY-Index-Control header which can be used by clients
to disallow yacy to index the response that belongs to the request where 
   X-YACY-Index-Contro is set to "no-index"

*) Bugfix for Seed-List download via Remote Proxy.
   Now the pragma and cache-control http headers of the request are properly set to "no-cache" 
   See: http://www.yacy-forum.de/viewtopic.php?p=11639#11639

*) Bugfix for http-Proxy
   yacy has ignored "no-cache"- pragma and cache-control http headers that were send in requests.
   Now, these request headers are evaluated properly

TODO: Missing evaluation of "no-store" request headers

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@971 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-10-23 10:35:05 +00:00
theli
539f9e57aa *) Disabling RemoteProxy if remoteProxyName is null or empty
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@970 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-10-23 09:03:40 +00:00
theli
5a25ad9109 *) Bugfix for useRemoteProxy4YACY and useRemoteProxy4SSL check
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@969 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-10-23 08:59:11 +00:00
theli
02d9af1a70 *) Restructuring and extending of Remote Proxy Support
- remote proxy configuration can now be "really" changed on the fly and takes effect immediately
   - adding possibility to disable remote proxy usage for yacy->yacy communication
   - adding possibility to disable remote proxy usage for ssl
   - restructuring proxy configuration so that it is stored in a single place now

*) Adding possibility to import a foreign word DB (or even more of them in parallel) 
   at runtime into the peers DB
   - this can be done by calling IndexImport_p.html 
   - ATTENTION: please not that at the moment this thread must be aborted via gui
     before a normal server shutdown is done. 
   - TODO: integrating IndexImport Thread into normal server shutdown
   - TODO: Adding posibility to import crawl-queues, etc. from foreign peers
   - TODO: removing old import function from yacy.java and calling the new routines instead

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@968 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-10-22 13:28:04 +00:00
low012
222607ef0f *) added <pre>-tag for preformatted text to yacyWiki (as requested by theli)
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@967 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-10-22 13:00:36 +00:00
allo
117a424d00 bugfix for sharing png/gif files in WWW/SHARE
http://www.yacy-forum.de/viewtopic.php?p=11565


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@966 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-10-20 14:58:15 +00:00
theli
af3060938b *) Bugfix for manual peer ping functionality
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@965 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-10-20 11:43:43 +00:00
theli
6e3201b74d *) Bugfix in httpc.java
- Requestheader was not passed to the underlying post function properly
   - Bug seems not to have caused any side-effect until yet

*) Bugfix for manual peer ping functionality

*) Bugfix for UnresolvedPattern Problem if an Exception occurred in a servlet.
   See: http://www.yacy-forum.de/viewtopic.php?t=1353

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@963 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-10-20 09:55:12 +00:00
borg-0300
58b670201d now, changed HTCacheSize needs no restart
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@961 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-10-19 17:59:54 +00:00
allo
f97c303ebd rights for Admin and Proxy.
Adminrights are OR(old auth or new).
Proxyrights are AND(you need Proxyrights and a not reached Timelimit)


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@960 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-10-19 12:20:08 +00:00
allo
97de600a68 another bugfix for share/www.
Now you can Use share/ and not only share/dir.html


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@958 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-10-19 11:42:15 +00:00
allo
df147d0fb0 rightsmanagement for upload/download User - preparations for proxy and admin User
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@957 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-10-19 11:26:35 +00:00
allo
2dfd6bf36a fix for networkimage
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@956 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-10-19 10:25:56 +00:00
allo
ec10220d57 Fix for last Commit: .class Files in htroot, not in the dir of the localized HTML-Files
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@955 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-10-19 07:17:49 +00:00
allo
4db2080188 Bugfix for www and share.
http://www.yacy-forum.de/viewtopic.php?p=11486


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@954 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-10-19 06:52:43 +00:00
theli
40777556c5 *) Connection Tracking
- adding automatic refresh
   - accepts new parameter nameLookup which can be used to deactivate 
     yacy-peer name lookup (because we have problems with this on large seed-dbs)

*) ViewFile
   New page that can be used to view 
   - original content 
   - plain text content 
   - parsed content
   - parsed sentences 
   of a webpage specified by there url hash
   Mainly for debugging purpose at the moment

*) Robots.txt 
   Bugfix for if-modified-since usage
   TODO: synchronization of downloads to avoid loading the same robots-file 
   multiple times in parallel by different threads

*) Shutdown
   Better abortion of transferRWI and transferURL sessions on server shutdown

*) Status Page
   Adding icon to start/stop crawling via status page

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@950 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-10-18 07:45:27 +00:00
borg-0300
bcb0d6d5ff changed setLastSeen(long rd) to setLastSeen();
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@949 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-10-17 23:23:17 +00:00
borg-0300
cdbaf637fb added
- getIP(), getJunior(), getSenior(), getPrincipal(); 
- setIP(), setJunior(), setSenior(), setPrincipal(), setLastSeen(long rd); 
- isPeerOK(), isOnline(String type);
next try to remove hello.class java.util.ConcurrentModificationException:null ;)

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@948 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-10-17 23:12:40 +00:00
borg-0300
e642a5d8b7 more constants
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@947 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-10-17 15:46:12 +00:00
borg-0300
7b510b558a small change
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@946 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-10-17 13:59:50 +00:00
borg-0300
ee7d847869 small change
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@945 6c8d7289-2bf4-0310-a012-ef5d649a1542
2005-10-17 13:57:54 +00:00