Marown/yacy_search_server
1
0
Fork 0
mirror of https://github.com/yacy/yacy_search_server.git synced 2024-09-21 00:00:13 +02:00
Code Issues Actions Packages Projects Releases Wiki Activity
7,722 Commits 1 Branch 20 Tags 836 MiB
2236e01137
Commit Graph

4671 Commits

Author SHA1 Message Date
cominch
2236e01137 Minor correction to prevent useless comma at beginning of string, created from list 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8059 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-11-20 15:20:57 +00:00
apfelmaennchen
5581be12fb YMarks: 
- added backend and api for tag management


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8058 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-11-19 20:59:21 +00:00
apfelmaennchen
a3eebfdcba YMarks: 
- show active/running crawls
- execute crawls (works currently only if API entry is available)
- various smaller fixes

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8056 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-11-17 23:11:27 +00:00
orbiter
c50f8f9a06 code cleanup 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8055 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-11-17 13:40:22 +00:00
apfelmaennchen
4f95f72124 YMarks: 
- working direct importer for YaCy Crawl Starts
- working direct import for old bookmarks.db

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8052 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-11-16 23:10:53 +00:00
orbiter
aa322bc6d0 fix 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8050 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-11-16 15:36:30 +00:00
orbiter
97d1347adb added also a default accept field to robots.txt downloads 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8049 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-11-16 15:33:55 +00:00
orbiter
f183d3822c added a default accept header in http requests since some http fraud detection functions check that this header field exist 
see also: http://bad-behavior.ioerror.us/ in source file browser.inc.php

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8048 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-11-16 15:27:43 +00:00
orbiter
06352b8d6b more logging 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8047 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-11-16 14:09:50 +00:00
orbiter
a99934226e more logging for debugging of robots.txt 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8046 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-11-16 13:56:31 +00:00
orbiter
7a5841e061 fix for robot parser 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8045 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-11-16 13:12:46 +00:00
orbiter
458c20ff72 fix for robot parser 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8044 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-11-16 13:06:46 +00:00
orbiter
017a01714d - enhanced logging in robots.txt parser for remote debugging 
- robots.txt is now more robust against database operations

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8043 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-11-16 01:03:49 +00:00
apfelmaennchen
a8dfe787ed - updated to jquery flexigrid 1.1 
- YMarks.html automatically  recognizes if a bookmark is a crawl start


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8040 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-11-15 21:45:17 +00:00
orbiter
eb1c7c041d write info about robots.txt evaluation into getpageinfo_p.xml 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8038 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-11-15 00:33:54 +00:00
apfelmaennchen
abba31f02e - bugfix for correctly sorting ymarks 
- some tuning for the autotagger (still not perfect)
- /api/ymarks/get_metadata.xml now provides info for crawlstarts
- removed unused code

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8036 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-11-14 22:00:44 +00:00
orbiter
775b44017e refactoring 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8033 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-11-14 15:11:57 +00:00
apfelmaennchen
5f7dbe1c42 - some refactoring (ymarks) 
- improvement for autotagger (is now able to create/detect  multi word tags e.g. 'open source')



git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8031 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-11-13 23:19:47 +00:00
orbiter
78ce3b13be typo 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8027 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-11-10 11:57:26 +00:00
orbiter
85d6bf4ac4 fixed urls to media content during indexing 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8021 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-11-09 15:40:14 +00:00
orbiter
0d858d48ec replaced String with StringBuilder in suggestion process 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8020 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-11-09 14:42:55 +00:00
orbiter
3a807e10cf - added a cache for active crawl profiles to the crawl switchboard 
- moved the domain cache for domain counter from the crawl switchboard to the crawl profiles. the crawl domain counter is now therefore relative for each crawl start, not for the whole crawler.

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8018 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-11-08 15:38:08 +00:00
orbiter
37e35f2741 normalization of url using urlencoding/decoding 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8017 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-11-08 12:02:22 +00:00
orbiter
1b86d06d1e fix for http://bugs.yacy.net/view.php?id=62 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8004 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-10-26 10:07:16 +00:00
orbiter
9e4875230f performance hacks 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8001 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-10-20 23:06:49 +00:00
orbiter
a9838f8b99 fix for http://bugs.yacy.net/view.php?id=59 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7997 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-10-12 22:26:48 +00:00
orbiter
a7df70221e refactoring 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7987 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-10-04 09:06:24 +00:00
orbiter
cf4fd525ee added directDocByURL attribute in crawl profile 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7985 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-09-30 12:38:28 +00:00
orbiter
c61e4cfd78 - fix for incomplete clear() in balancer 
- renamed Parser Errors to Rejected URLs

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7984 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-09-30 10:27:14 +00:00
orbiter
813f297a95 another performance hack: re-use of known host addresses for isLocal property; avoids look-up in local hash 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7983 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-09-30 08:26:31 +00:00
orbiter
035ebfbf3b - performance hacks (should affect the crawl balancer and reduce CPU load during crawl stack re-fill) 
- this may have also (good) performance side effects on other parts of YaCy


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7982 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-09-30 07:57:50 +00:00
orbiter
b250e6466d implemented crawl restrictions for IP pattern and country lists 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7980 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-09-29 15:17:39 +00:00
f1ori
e207c41c8e * fix urlproxy for urls containing dolar signs 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7979 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-09-29 12:53:55 +00:00
orbiter
5ad7f9612b added crawl settings for three new filters for each crawl: 
must-match for IPs (IPs that are known after DNS resolving for each URL in the crawl queue)
must-not-match for IPs
must-match against a list of country codes (allows only loading from hosts that are hostet in given countries)

note: the settings and input environment is there with that commit, but the values are not yet evaluated

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7976 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-09-27 21:58:18 +00:00
orbiter
d2ea250d99 refactoring: 
- moved many classes from de.anomic to net.yacy
- made more sub-packages for search classes

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7973 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-09-25 16:59:06 +00:00
low012
42b5f09f68 *) this should fix a bug in snippet creation (also cleaned up a little bit) 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7972 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-09-25 16:07:22 +00:00
orbiter
6b22865dbc - removed some warinings 
- removed a dead update location

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7970 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-09-24 01:58:54 +00:00
orbiter
0c6d95e57b - more tolerance against failure of table opening 
- more connections for solrj

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7968 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-09-21 15:08:05 +00:00
orbiter
4f31869c5a enhanced search result timing 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7966 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-09-21 10:43:08 +00:00
orbiter
6b02b696b0 - add number of search results to end of rss and json output to reflect latest status of retrieval 
- distinguish search access with different verify state in access of search cache

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7965 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-09-20 19:41:44 +00:00
f1ori
87e6abd168 * fix urls containing a port number in urlproxy 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7964 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-09-20 15:02:15 +00:00
f1ori
97045022fa * pass cookies to Server Side Includes 
* User.html a bit more usable


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7963 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-09-20 14:54:14 +00:00
orbiter
ce2a76d603 performance hack for search process 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7961 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-09-16 10:00:51 +00:00
orbiter
2c4a672fe2 bugfixes and performance hacks for tabe index 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7957 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-09-15 11:17:02 +00:00
orbiter
dad5b586a4 added a concurrent warmin-up of Table data structures. that should speed-up the start-up process but may also cause stronger CPU load at that time. 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7956 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-09-15 10:01:21 +00:00
orbiter
734059d33e performance hacks 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7955 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-09-14 23:34:05 +00:00
orbiter
23e81b28b2 synchronization enhancements 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7954 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-09-14 21:19:02 +00:00
orbiter
dd4635e323 patches 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7953 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-09-14 20:11:27 +00:00
orbiter
bb0c045036 fix for problem with relocation of network 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7944 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-09-13 18:46:11 +00:00
orbiter
85a5487d6d YaCy can now use the solr index to compute text snippets. This makes search result preparation MUCH faster because no document fetching and parsing is necessary any more. 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7943 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-09-13 14:39:41 +00:00