Marown/yacy_search_server
1
0
Fork 0
mirror of https://github.com/yacy/yacy_search_server.git synced 2024-09-21 00:00:13 +02:00
Code Issues Actions Packages Projects Releases Wiki Activity
7,111 Commits 1 Branch 20 Tags 836 MiB
30df013c2b
Commit Graph

483 Commits

Author SHA1 Message Date
f1ori
a321c7673d * adminAccountForLocalhost only for localhost 
* yacy crawls local domains also, if no password is set (the interface is already protected)
* it's not required anymore, to set a password in intranet mode

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7436 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-01-17 11:37:30 +00:00
hermens
930cb412dd Let SHORT_MILSEC_FORMATTER make a new formatted String every millisecond 
see http://forum.yacy-websuche.de/viewtopic.php?f=6&t=3103


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7434 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-01-15 13:16:40 +00:00
orbiter
090c73e32e catch a OOM in HeapReader iteration 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7433 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-01-12 12:04:18 +00:00
low012
48463c4507 *) General private License? ;-) 
*) minor code changes

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7432 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-01-12 00:28:08 +00:00
orbiter
6c1b14c8e1 - more control in access tracker: count number of returned search results (not only info how much is in the index) 
- extended query params for this
- enhanced cora

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7430 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-01-11 22:58:14 +00:00
low012
9f38c0023d *) Minor changes, mainly cleaning up a little bit, no functional changes. 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7428 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-01-10 20:24:52 +00:00
orbiter
54e77e6255 refactoring 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7426 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-01-10 08:40:41 +00:00
orbiter
feefe17568 npe assert fix 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7424 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-01-04 12:24:41 +00:00
orbiter
733903f2c9 fix for http://forum.yacy-websuche.de/viewtopic.php?p=21489#p21489 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7422 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-01-03 23:06:33 +00:00
orbiter
24e4126eee added JSON parser code from json.org (added generics to it) 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7421 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-01-03 22:53:15 +00:00
orbiter
10ae8d961b - cora package has now no dependencies to other yacy packages and becomes a 'base' package (refactoring) 
- cleaned up (removed special code and documentation for 27c3)
- added remote search functions to be used within cora

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7420 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-01-03 20:52:54 +00:00
lotus
0e54233408 UPnP: map port again if we are not reachable (e.g. when router rebooted) 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7419 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-01-02 21:17:21 +00:00
lotus
b1484299b2 same units for memory observer configuration (MiB) 
old setting for DHT (RAM) will be lost after update
can be set on /Performance_p.html

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7418 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-01-02 20:38:01 +00:00
orbiter
387db84087 maybe found bug in non-working index dumper 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7414 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2010-12-30 00:02:42 +00:00
orbiter
a4c9d27287 - moved some variables from Stwitchboard to new class AccessTracker 
- added a limitation in access tracking to delete queries which are older than 10 minutes

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7410 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2010-12-29 01:54:27 +00:00
f1ori
e4aabaa1c3 * fix negative filelength for files >2G 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7408 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2010-12-28 17:25:39 +00:00
orbiter
cdfe8afe3f fix for really bad table iteration implementation: reduction of IO 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7407 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2010-12-28 16:44:55 +00:00
low012
9eae33f886 *) Ooops... 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7406 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2010-12-28 13:04:48 +00:00
low012
a001e8075c *) minor enhancements 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7405 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2010-12-28 13:03:49 +00:00
low012
11ea966f9e *) added SID file (Commodore 64) sound file parser 
*) minor changes

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7403 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2010-12-28 12:06:04 +00:00
orbiter
b2ed4cfaf8 more small bugfixes and light refactoring 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7401 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2010-12-28 01:57:05 +00:00
orbiter
3ca06d6290 patch for http://forum.yacy-websuche.de/viewtopic.php?p=21460#p21460 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7399 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2010-12-27 23:57:29 +00:00
orbiter
903c824c2c - allow only scanned resourced with granted status 
- increased time-out when scanning an ip range

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7398 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2010-12-27 23:11:56 +00:00
low012
936e976c23 *) added FreeMind (http://freemind.sourceforge.net/) mindmap parser 
*) minor changes

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7397 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2010-12-27 20:13:31 +00:00
low012
3d95981f7d *) cleaning up the code a little bit 
*) minor changes

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7396 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2010-12-27 17:07:21 +00:00
low012
2a6499364d *) minor changes 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7395 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2010-12-27 15:53:41 +00:00
low012
c0274bd123 *) minor changes 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7394 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2010-12-27 15:37:11 +00:00
orbiter
fe46536f6e enhanced network scanner (less name resolving during scanning and no name resolving during search) 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7392 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2010-12-26 16:25:17 +00:00
orbiter
e753027c43 fix for http://forum.yacy-websuche.de/viewtopic.php?p=21439#p21439 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7390 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2010-12-22 00:04:30 +00:00
orbiter
bf4ef1513e - fix for map view 
- remove some UNRESOLVED PATTERN
- maybe a fix for non-flushing cache

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7389 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2010-12-21 23:48:05 +00:00
orbiter
6b70393d1d - new java version 1.6 
- replaced old gif animator by java 1.6 gif animator

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7388 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2010-12-20 22:51:50 +00:00
orbiter
e88c428008 fix to ftp loader 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7387 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2010-12-18 10:22:54 +00:00
orbiter
59b70a5a92 another fix to the ftp crawler: now correct directory listings according to rfc2640 (path with spaces) and better title names for such files 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7386 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2010-12-18 00:44:19 +00:00
orbiter
9b25a33fd9 - fixed numerous bugs 
- better document names
- fixed problem with ftp crawling
- added automatic removal of search results from services that are not online according to the latest network scan: this does not delete the index but just does not show them. after the next network scan when the server is available again, the results are again showed.

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7385 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2010-12-17 17:30:09 +00:00
orbiter
7bdb13bf7f more fixes to smb crawling: better file names 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7384 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2010-12-17 00:52:24 +00:00
orbiter
94c48500cc several fixes 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7383 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2010-12-17 00:11:42 +00:00
orbiter
0ac7311a62 fix for token parser 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7382 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2010-12-16 23:47:36 +00:00
orbiter
58b59f9bc8 - a collection of bug fixes and some redesign of the Scanner class 
- fixed smb crawling
- added smbget to download script generation

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7381 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2010-12-16 23:37:21 +00:00
orbiter
c288fcf634 redesigned CrawlStartScanner user interface and added more features: 
- multiple hosts for environment scans can be given (comma-separated)
- each service (ftp, smb, http, https) for the scan can be selected
- the scan result can be accumulated or refreshed each time a network scan is made
- a scheduler was added to repeat a scan and add all found urls to the indexer automatically

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7378 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2010-12-16 02:15:20 +00:00
f1ori
9d2159582f * fix system update if urls are in blacklist (for example for very general blacklists like *.de) 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7375 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2010-12-15 19:20:00 +00:00
orbiter
56264dcc17 - added CamelCase parser to MultiProtocolURI: generate better to-be-indexed words from urls 
- integrated new parser into loader processes: enrich document parser
- fixed a concurrent modification exception in kelondro iterator
- hand-over of document size from crawler to indexer

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7374 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2010-12-15 00:03:19 +00:00
orbiter
99a7fe87f9 - removed old intranet scanner (the generic scanner now completely subsumes the old one) 
- added information about granted access
- enhanced servlet design
- added submit-feedback (because it is a long-running task)

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7372 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2010-12-14 01:14:15 +00:00
orbiter
acab6801d9 added new network scanner 
- you can scan any ip or host in the internet for services
- this replaces the intranet scanner

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7371 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2010-12-13 18:19:37 +00:00
orbiter
14e4fae8e9 fixes to ftp client 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7369 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2010-12-11 09:21:37 +00:00
orbiter
a563b05b60 enhanced crawler: 
- added a new queue 'noload' which can be filled with urls where it is already known that the content cannot be loaded. This may be because there is no parser available or the file is too big
- the noload queue is emptied with the parser process which indexes the file names only
- the 'start from file' functionality now also reads from ftp crawler

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7368 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2010-12-11 00:31:57 +00:00
orbiter
c36da90261 added a very fast ftp file list generator to site crawler: 
- when a site-crawl for ftp sites is now started, then a special directory-tree harvester gets the complete directory structure of a ftp server at once
- the harvester runs concurrently and feeds into the normal crawl queue

also in this:
- fixed the 'start from file' crawl function
- added a link detector for the html parser. The html parser can now also extract links that are not included in <a> tags.
- this causes that a crawl start is now also possible from clear text link files

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7367 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2010-12-09 17:17:25 +00:00
orbiter
db99db4be9 some redesign of the search-fail-response mechanism: 
when a search fails for a single url because the snippet cannot be generated, then the url reference is deleted from the index. This mechanism was redesign and enhanced. The process now also writes into the work tables into the table searchfl to prepare a re-indexing mechanism.

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7364 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2010-12-06 14:34:58 +00:00
f1ori
4915d1781a * use local backup-file, if remote network-definition is not availible 
* resolve single point of failure in networks, managed by central network-definitions

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7363 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2010-12-06 11:09:16 +00:00
orbiter
4e2c14efbb fixed bugs in parser and ftp client 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7360 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2010-12-02 11:05:04 +00:00
orbiter
d78e322e84 added a directory-structure reader to ftp client 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7359 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2010-12-02 08:08:01 +00:00