Marown/yacy_search_server
1
0
Fork 0
mirror of https://github.com/yacy/yacy_search_server.git synced 2024-09-21 00:00:13 +02:00
Code Issues Actions Packages Projects Releases Wiki Activity
4,867 Commits 1 Branch 20 Tags 836 MiB
5b2a57bfd0
Commit Graph

3119 Commits

Author SHA1 Message Date
apfelmaennchen
5b2a57bfd0 - /xml/util/getpageinfo_p.xml added <desc> and <lang> tags 
- changed htmlFilterContentScraper.getKeywords() to split either space or comma charater not both

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5183 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-09-18 21:01:23 +00:00
orbiter
e1f67262f7 - added and removed some debugging output 
- fixed a bug with merge method
- patched wrong output of language identification (not fixed, only patched!)

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5181 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-09-18 14:12:15 +00:00
orbiter
ce2a7ed116 integrated language detection classes into condenser environment 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5180 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-09-18 13:12:33 +00:00
orbiter
2b13705839 fixed a mistake in indexing queue processing: documents had been parsed before it was checked if they should be indexed or not. parsing was not necessary for this check, so the check was moved in the queue in front of the document parsing 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5179 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-09-18 11:36:09 +00:00
orbiter
21dbb39afa switched two balancer cases 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5177 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-09-17 22:13:25 +00:00
orbiter
1bbf362cef update to the crawl balancer: better organization and better crawl delay prediction 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5176 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-09-17 21:45:21 +00:00
orbiter
ddcf285499 - fixed a bug in performance setting (did not work with german translation) 
- reduced maximum number of error url references to save some memory (this was actually a small memory leak)

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5174 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-09-16 23:04:24 +00:00
orbiter
0cd0fee546 fixed bug with wrong proxy result enqueueing. See: 
http://forum.yacy-websuche.de/viewtopic.php?p=8130#p8130
- removed the online status property. This influenced the proxy behavior and created some complexity that was not needed because the online status was never used as it was ceated for (offline browsing)
- checked all proxy identification procedures during crawling and enhanced transparency and error checking
- fixed a proxy identification routine that caused the wrong selection of the proxy result queue

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5173 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-09-16 21:56:23 +00:00
orbiter
670244849d fix for http://forum.yacy-websuche.de/viewtopic.php?p=9835#p9835 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5164 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-09-15 18:29:37 +00:00
lotus
fd9233244e configurable free disk space via disk.free 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5163 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-09-15 17:33:06 +00:00
orbiter
25a62cdc3f small fixes 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5161 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-09-15 15:11:59 +00:00
lotus
73f233bb11 * set resource observer to 1000MB 
* transparent favicon

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5160 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-09-15 12:41:27 +00:00
orbiter
5fbccfd75e fix for http://forum.yacy-websuche.de/viewtopic.php?f=6&t=1366&p=9348#p9348 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5155 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-09-14 20:10:43 +00:00
orbiter
a28faabfd2 fix for http://forum.yacy-websuche.de/viewtopic.php?f=6&t=1351&p=9242#p9242 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5154 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-09-14 20:03:59 +00:00
apfelmaennchen
7b63c66a08 - bugfix in bookmarksDB.Tag.hasPublicItems() 
- this anoying little bug prevented display of public items without admin login for /xml/bookmarks/...

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5151 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-09-14 18:45:08 +00:00
orbiter
1fb1665e71 increased dht interval to avoid peer selection failure 
(maybe too less peers available to fill the big gaps)

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5143 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-09-12 13:38:27 +00:00
orbiter
1eb813bd43 shifted index deletion-on-exit rule to the class where the errors are produced 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5141 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-09-12 11:51:48 +00:00
f1ori
ba76995d2c * fix for http://forum.yacy-websuche.de/viewtopic.php?f=6&t=1415 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5140 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-09-12 10:54:11 +00:00
f1ori
bea6c13139 * with r5137 robotParser didn't work at all -> fix 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5139 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-09-12 09:06:38 +00:00
lotus
3ded1efe84 kelondroExceptionCounter didn't work 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5138 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-09-11 18:51:47 +00:00
f1ori
ae677e1738 * fix problem in robotparser, see http://forum.yacy-websuche.de/viewtopic.php?f=5&t=1421&p=9742 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5137 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-09-11 18:12:17 +00:00
lotus
383d89481e count errors before deleting collection.index 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5136 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-09-10 16:40:20 +00:00
lotus
0bb4fbc403 delete corrupted collecion.index on exit for rebuild on next start 
see http://forum.yacy-websuche.de/viewtopic.php?p=9725#p9725

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5135 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-09-10 12:55:14 +00:00
lotus
b68d06a6e8 performance settings based on network's remote crawl speed 
removed some _pro values from config

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5134 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-09-10 12:52:17 +00:00
danielr
d60b2b198d proxy fixed 'not modified' http://forum.yacy-websuche.de/viewtopic.php?f=6&t=1419 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5133 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-09-10 11:06:22 +00:00
f1ori
bd0318ba81 * YaCy only supports gzip-encoding, so remove any other encoding from request 
* fixes http://www.yacy-forum.org/viewtopic.php?f=2&t=163


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5132 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-09-09 14:04:52 +00:00
orbiter
bb5c898441 enhancements to localsearch behavior 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5131 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-09-09 10:24:42 +00:00
orbiter
42e2d195ac added hint from http://forum.yacy-websuche.de/viewtopic.php?f=6&t=1294 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5130 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-09-08 22:37:58 +00:00
orbiter
39964e88fa fix for http://forum.yacy-websuche.de/viewtopic.php?f=6&t=1329#p9121 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5129 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-09-08 22:06:45 +00:00
orbiter
3f3673b6e5 extended balancer: 
- added automatic time delay in case that a large number of urls come from the same domain
- added additional time delay in case that an url is a dynamic (CGI) url. This shall cause less IO on targets


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5128 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-09-08 21:50:37 +00:00
orbiter
3c6e8d2015 set default ppm when network is switched 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5127 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-09-08 18:20:05 +00:00
orbiter
3288c19c1a reduce remote crawl PPM for fresh peers in freeworld to 6 PPM 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5124 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-09-08 09:49:08 +00:00
lotus
5ce9a100bb fix(2) for http://forum.yacy-websuche.de/viewtopic.php?f=6&t=1416 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5122 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-09-07 13:57:53 +00:00
danielr
cf29ca19d4 possible fix for POST character encoding http://forum.yacy-websuche.de/viewtopic.php?f=5&t=1374 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5121 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-09-07 13:10:46 +00:00
danielr
a2eeb6138c fix for http://forum.yacy-websuche.de/viewtopic.php?f=6&t=1416 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5120 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-09-07 13:04:17 +00:00
orbiter
d09ddabd09 corrected a design mistake (5-byte hashes not necessary) 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5119 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-09-04 21:28:00 +00:00
orbiter
c97d0fcee7 modified the domain list export function: 
- used the new superfast domain list generation from the domain statistics
- better interactive behavior

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5118 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-09-04 20:28:36 +00:00
orbiter
77ee0765a4 - added domain statistic generation to IndexControlURLs_p.html servlet 
- added 'delete all' button to all results of such a domain statistic output which causes that all urls to this domain are deleted
- extended stack cleaner to clean also the statistics: they are not completely destroyed, only the smallest counting domains are removed


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5117 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-09-04 19:41:57 +00:00
orbiter
80a7bc93d6 - added statistical evaluation about domains that appear during crawling 
- added tables that show this statistics in CrawlResults web pages

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5113 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-09-04 09:59:17 +00:00
orbiter
4fbee21cea - added fetch-ahead again (had been removed in last commit) 
- reverted default query mode to verify=false

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5111 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-09-03 23:50:13 +00:00
lotus
423a89ebe8 * fix if yacy was installed to a path with whitespace 
* show nice dots when waiting for restart/update

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5110 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-09-03 18:49:02 +00:00
orbiter
fc03b0437a fixed a error case where a second search after a first search with a different search word failed 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5109 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-09-03 15:55:25 +00:00
orbiter
eca171ba2e fix for case where javascript was not filtered by the html parser 
see http://forum.yacy-websuche.de/viewtopic.php?p=9667#p9667

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5108 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-09-03 14:41:20 +00:00
lotus
e645bae29f display table in log 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5106 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-09-03 13:14:01 +00:00
orbiter
ead39064c5 fixed problem with wrong result number calculation 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5105 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-09-03 10:04:46 +00:00
hermens
2437beb96c fix for http://forum.yacy-websuche.de/viewtopic.php?f=6&t=1360&p=9321#p9321 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5104 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-09-03 07:39:03 +00:00
orbiter
7b12e77a63 fix for http://forum.yacy-websuche.de/viewtopic.php?f=5&t=1393&hilit=&p=9655#p9655 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5103 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-09-03 00:50:42 +00:00
orbiter
05dbba4bab added logging conditions to all fine and finest log line calls 
this will prevent an overhead for the generation of the log lines in case that they then are not printed

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5102 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-09-03 00:30:21 +00:00
orbiter
d3d41e2ee4 - fixed problem with searching with quotes (still not complete, but not as bad as before) 
- fixed parsing of crawl-delay statements when seconds were given with float numbers
- enhanced performance of profiling (not too many loggings; not more than one per second)
- removed some debug output
- fixed wrong return type in logging
- added a logging condition in httpd to prevent that logging statements are generated when they are not written (should be added everywhere!)
- fixed wrong word distance computation in RWI management


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5101 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-09-02 23:49:48 +00:00
lotus
3fbfd5a78b * fix for non-changing offset on new search term 
* dht-heap doesn't has to be deleted (5097), we simply write a new one on exit
* do not install YaCy in startup because a Windows-shutdown might corrupt something. Installing YaCy as a service would solve this.

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5099 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-09-02 15:09:31 +00:00