Marown/yacy_search_server
1
0
Fork 0
mirror of https://github.com/yacy/yacy_search_server.git synced 2024-09-21 00:00:13 +02:00
Code Issues Actions Packages Projects Releases Wiki Activity
1,018 Commits 1 Branch 20 Tags 836 MiB
93cadb47b9
Commit Graph

714 Commits

Author SHA1 Message Date
theli
93cadb47b9 *) More tolerant robots parser for robots-files which missing empty lines between rule blocks 
See: http://www.yacy-forum.de/viewtopic.php?p=12471

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1048 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-11-08 07:41:25 +00:00
orbiter
02f8013013 auto-delete of corrupted word files during word-migration 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1047 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-11-07 14:57:37 +00:00
orbiter
d2731418bf added creation of global ranking files and changed url normal form usage 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1046 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-11-07 12:33:02 +00:00
theli
6f9f8ed8f8 *) Automatic Reset of Stack Crawler DB on startup errors 
See: http://www.yacy-forum.de/viewtopic.php?t=1432

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1045 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-11-07 12:19:05 +00:00
theli
fb766413d1 *) Changes on httpc dns caching 
- Bugfix: old dns cache did not handle case insensitive hostnames correctly. 
   - adding a possibility to set domain name patterns defining hostnames that should not be cached by the httpc dns cache
     e.g. borg-300.dyndns.org
     This can be done by setting the new httpc.nameCacheNoCachingPatterns property
   - using httpc.dnsResolve wherever possible within the sourcecode
     [httpd.java,plasmaCrawlStacker.java]

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1044 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-11-07 10:57:54 +00:00
orbiter
bc420c62f6 fixed htcache path generation (never change a running system) 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1041 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-11-07 01:31:11 +00:00
borg-0300
795f488222 new urlNormalform version 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1040 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-11-06 22:35:56 +00:00
orbiter
c86d801b0f removed dyndns domains from dns caching 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1039 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-11-06 22:12:08 +00:00
orbiter
6dc42a2392 detecting of loops in kelondroTree during last/first-Node search 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1038 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-11-06 21:06:55 +00:00
borg-0300
17d2830394 see: http://www.yacy-forum.de/viewtopic.php?t=1416 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1037 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-11-06 17:24:57 +00:00
theli
dd24f0252f *) Searchword highlighting for info page 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1036 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-11-06 06:27:17 +00:00
theli
f9fb284fb7 *) Better handling of robots.txt files with incorrect keywords 
See: http://www.yacy-forum.de/viewtopic.php?p=12292#12292

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1035 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-11-06 06:01:08 +00:00
borg-0300
a1406f4617 urlNormalform: no logging 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1034 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-11-05 16:21:04 +00:00
borg-0300
72cde1d894 getCachePath: no logging 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1033 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-11-04 22:47:13 +00:00
borg-0300
1fbd72f9e0 rename "index.html" to "ndx" 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1032 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-11-04 22:39:33 +00:00
borg-0300
cd1107d85e added support for URLs with '?&' 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1030 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-11-04 17:25:15 +00:00
borg-0300
5fb2b017cb small change 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1029 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-11-04 16:37:56 +00:00
borg-0300
60e869f236 bugfix for last commit 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1028 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-11-04 15:52:12 +00:00
borg-0300
544e4ea90e small change 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1027 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-11-04 14:11:46 +00:00
borg-0300
00ab4d8723 cleaned, small change, Properties 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1026 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-11-04 13:41:51 +00:00
theli
b8ceb1ffde *) Adding better https support for crawler 
- solving problems with unkown certificates by implementing a dummy trust Manager
   - adding https support to robots-parser 
   - Seed File can now be downloaded from https resources
   - adapting plasmaHTCache.java to support https URLs properly

*) URL Normalization
   - sub URLs are now normalized properly during indexing
   - pointing urlNormalForm function of plasmaParser to htmlFilterContentScraper function
   - normalizing URLs which were received by a crawlOrder request

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1024 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-11-03 15:28:37 +00:00
borg-0300
d2507c6081 rename setJunior()... to orJunior()..., 
added javadoc, 
added getPeerType(), setIP(), setPort(String port)


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1023 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-11-03 14:16:16 +00:00
borg-0300
e3179a6394 added getOwnSeedFile() 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1022 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-11-03 14:07:58 +00:00
borg-0300
a803a509ae bugfix: port handling in HTCache 
grogram flow, cleared up


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1021 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-11-03 12:39:24 +00:00
theli
3d0dfd4df4 *) Using StringBuffer instead of String concatenation 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1018 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-11-03 08:08:37 +00:00
low012
452db479cd *) bugfix: "21" was displayed as "21" in yacyWiki 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1017 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-11-02 23:28:54 +00:00
hydrox
cb69047b91 *)cleanup access static methods and fields 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1016 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-11-02 17:56:26 +00:00
hydrox
56b9f34411 *)removed unused imports 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1015 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-11-02 16:30:45 +00:00
orbiter
5f68b6886b introduced new url-hashes for better ranking computation 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1013 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-11-02 00:54:55 +00:00
orbiter
aadace1285 fixed network image in search performance monitor 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1012 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-11-01 00:49:13 +00:00
orbiter
bb369c98de fixed search result ordering by date 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1011 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-10-31 17:17:48 +00:00
hydrox
295aff52a3 *)added offline-browsing-support (onlineMode=0) 
*)online-mode now can be changed in Status.html

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1010 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-10-31 12:25:40 +00:00
orbiter
4d1e56e4d9 fixed intermission-bug (removed 'break for intermission' of httpd-thread) 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1009 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-10-31 10:46:13 +00:00
orbiter
b058ecf0bc refactoring of image-generation; added experimental PNG encoder (not active now) 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1008 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-10-31 02:43:55 +00:00
orbiter
7a9fc653e9 fixed 100% cpu bug in wikiCode 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1007 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-10-30 14:27:09 +00:00
orbiter
1aa707d357 fix for http://www.yacy-forum.de/viewtopic.php?p=11866#11866 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1005 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-10-29 15:30:06 +00:00
orbiter
d42531e1b2 added auto-reset for NURL-DBs 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1004 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-10-29 13:53:32 +00:00
allo
5526d2ff73 Basic WatchCrawler.html, which loades the Queues using AJAX. 
ajax.js: generic Ajax-Functions
xml/queues/indexing_p*: the Backend


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1002 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-10-29 09:47:30 +00:00
allo
92c49b406b adminAuth with userDB and adminAuthenticated (fix for statuspage) 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1001 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-10-28 18:06:36 +00:00
rramthun
27f180f24b Update of YaWoStat to 0.2. 
Now does not try to make 400000! operations to load a 4MB textfile :-/

Program is not finished yet.

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1000 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-10-28 16:27:49 +00:00
theli
3b5d0eb053 *) Synchronizing robots.txt downloads to avoid parallel downloads of the same file by separate threads 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@998 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-10-28 09:10:48 +00:00
theli
723e056c48 *) Bugfix for ClassCastException during SessionPool.close 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@996 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-10-28 07:25:22 +00:00
theli
e58e85363d *) Bugfix for ConcurrentModificationException while operating on seed properties 
*) Bugfix for YACY database inconsistency (no more elements available in db '...seed.new.db'), re-set of db.
   See: http://www.yacy-forum.de/viewtopic.php?p=11836#11836
        http://www.yacy-forum.de/viewtopic.php?p=11814#11814

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@995 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-10-28 07:15:00 +00:00
orbiter
2dc12c45cb small corrections to profiling generation 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@994 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-10-28 01:29:54 +00:00
orbiter
d656e2b433 added a memory-profile chart generation to database performance testing 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@993 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-10-28 01:22:11 +00:00
theli
9319445f38 *) Minor changes to db performance test 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@992 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-10-27 11:37:21 +00:00
theli
ec3af327f7 *) Bugfix for Proxy-Authentication against remote proxy 
See: http://www.yacy-forum.de/viewtopic.php?p=11804#11804

*) Adding first version of db test for mysql
   NOTES:
   - db user + db + db table must be created before starting the test
   - db table must be empty. Entries can not be updated at the moment
   - db connection properties must be changed in the sourcecode at the moment
   TODOs:
   - accepting connection properties via command line
   - implementing update + remove + read operations
   - 'maybe' adding code to create db + table if it doesn't exists

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@991 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-10-27 11:28:37 +00:00
orbiter
5b0911d7ea added new performance menu for search sequence configuration and monitoring 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@990 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-10-27 00:42:08 +00:00
orbiter
4fa942511b de-serialized read and write access 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@989 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-10-26 23:05:20 +00:00
orbiter
09a0e898e0 created database test application 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@988 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-10-26 15:31:58 +00:00