Commit Graph

77 Commits

Author SHA1 Message Date
orbiter
306c50ac40 QPM (queries per minute) statistic stub
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3308 6c8d7289-2bf4-0310-a012-ef5d649a1542
2007-01-31 15:39:11 +00:00
allo
29aa7031d3 workaround for the snippets
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3225 6c8d7289-2bf4-0310-a012-ef5d649a1542
2007-01-16 21:35:25 +00:00
allo
8803f813c5 partly fixed snippets
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3224 6c8d7289-2bf4-0310-a012-ef5d649a1542
2007-01-16 21:18:32 +00:00
allo
0c81bd39d4 XSS-safe put as default.
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3217 6c8d7289-2bf4-0310-a012-ef5d649a1542
2007-01-16 14:07:54 +00:00
rramthun
00ca6ecf58 -made snippet-timeout for text and media configurable
-Now completely working OpenSearch plugin!

Please have a look at the search-field of modern browsers (IE 7+, FF2+). It should change its colour when you visit the index/search-page of a peer and you should be able to add your YaCy-peer as search source very easily now.
Credits for adapting the plugin to make it work go to Philipp Redeker.

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3212 6c8d7289-2bf4-0310-a012-ef5d649a1542
2007-01-15 19:45:15 +00:00
karlchenofhell
41bc31d2c2 - ConfigAdvanced_p => XHTML (no invalid IDs)
- removed unmappable characters from code

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3133 6c8d7289-2bf4-0310-a012-ef5d649a1542
2006-12-23 13:35:34 +00:00
orbiter
1d2d1854b9 added size of rwi and urls to WatchCrawler
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3112 6c8d7289-2bf4-0310-a012-ef5d649a1542
2006-12-21 21:33:35 +00:00
orbiter
0a050bc043 enhanced ranking
- redesign of data storage in plasmaSearchRankingProfile
- profiles are extended by new ranking parameters
- new RWI ranking parameters are considered during ranking
- appearance attributes (i.e. emphasised text) is now considered
- faster ranking
- some attributes that had been checked during post-ranking can now be
  checked during pre-ranking phase
- removed old ranking parameter on index.html page (will be replaced by profiles in the future)
- ranking can now consider appearances of media content
- snippet-loading for media types now work correctly (fetches only from the wanted media)
- ranking-profiles can be handed over the remote peers and apply there also
- re-search of same query with different domain now also re-triggers remote search

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3105 6c8d7289-2bf4-0310-a012-ef5d649a1542
2006-12-20 15:44:29 +00:00
orbiter
61798f0ae6 added option to distinguish between text crawl and media crawl
- for each crawl start, there is now a flag for text and media
- the localCrawl flag is superfluous
- added new crawl profiles
- if an image search is done, only media links are crawled for the snippets


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3100 6c8d7289-2bf4-0310-a012-ef5d649a1542
2006-12-19 03:10:46 +00:00
orbiter
febe6b114a design update of crawler monitor
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3094 6c8d7289-2bf4-0310-a012-ef5d649a1542
2006-12-18 01:18:28 +00:00
orbiter
e4570bffaf -implemented a specialized snippet-fetch for media content
-changed search result preparation for media search presentation

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3073 6c8d7289-2bf4-0310-a012-ef5d649a1542
2006-12-12 02:09:25 +00:00
orbiter
1377c53aa3 extraction of media links from search results
these links are mixed to the snippets for testing purpose
(a final version will handle this differently)

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3069 6c8d7289-2bf4-0310-a012-ef5d649a1542
2006-12-11 01:31:23 +00:00
orbiter
fb9e0f0284 preparations for media snippets
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3064 6c8d7289-2bf4-0310-a012-ef5d649a1542
2006-12-09 23:15:58 +00:00
orbiter
937ccd4e76 fix for snippet-generation
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3060 6c8d7289-2bf4-0310-a012-ef5d649a1542
2006-12-09 02:13:43 +00:00
orbiter
9a85f5abc3 cleanup
- removed 'deleteComplete' flag; this was used especially for WORDS indexes
- shifted methods from plasmaSwitchboard to plasmaWordIndex

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3051 6c8d7289-2bf4-0310-a012-ef5d649a1542
2006-12-06 12:51:46 +00:00
orbiter
109ed0a0bb - cleaned up code; removed methods to write the old data structures
- added an assortment importer. the old database structures can
  be imported with
  java -classpath classes yacy -migrateassortments
- modified wordmigration. The indexes from WORDS are now imported
  to the collection database. The call is
  java -classpath classes yacy -migratewords
  (as it was)

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3044 6c8d7289-2bf4-0310-a012-ef5d649a1542
2006-12-05 02:47:51 +00:00
orbiter
ceb9e3aa17 - enhanced parser: collection of audio, video, image and application links
- enhanced condenser: better handling of utf-8 and pre-formatted texts


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3017 6c8d7289-2bf4-0310-a012-ef5d649a1542
2006-11-28 15:00:15 +00:00
orbiter
b5a29e9651 - fix for snippets that are too short
- added keyword to snippet fetch to suppres removal of not-found snippet words (for debugging)


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3009 6c8d7289-2bf4-0310-a012-ef5d649a1542
2006-11-25 00:38:09 +00:00
orbiter
30888e7a2f implementation of search constraints
Such constraints may formulate specific restrictions to web searches
This is implemented by scraping information for constraints from a web
page during parsing, and storing flags to the pages within the web index.

In this first step, only information for index pages ("index of", directory listings)
are scraped and stored in flags
- added new flag class kelondroBitfield
- added scraper method in condenser
- added bitfield structure for all scrape types (see also condenser)
- added bitfield structure for appearance locations (see RWIEntry)
- added handover protocol for remote search and index distribution
- extended kelondroColumn class to hold bitfield types
- added another search attribute on search page (index.html)
- extended search-filter to enable filtering of non-matching constraints
- set all new database types to be default
- refactoring: moved word hash generation to condenser class

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2999 6c8d7289-2bf4-0310-a012-ef5d649a1542
2006-11-23 02:16:30 +00:00
orbiter
d34f10c63d some tests with reverse dns lookup
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2954 6c8d7289-2bf4-0310-a012-ef5d649a1542
2006-11-12 00:28:10 +00:00
orbiter
497428c8ec refactoring
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2949 6c8d7289-2bf4-0310-a012-ef5d649a1542
2006-11-10 01:13:33 +00:00
allo
a75f895884 memory and traffic informations
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2904 6c8d7289-2bf4-0310-a012-ef5d649a1542
2006-11-03 19:44:31 +00:00
allo
2ba56f70a8 XML-safe put.
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2848 6c8d7289-2bf4-0310-a012-ef5d649a1542
2006-10-23 10:35:48 +00:00
allo
a17c43779f removed wrong part of template
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2830 6c8d7289-2bf4-0310-a012-ef5d649a1542
2006-10-20 14:50:57 +00:00
allo
27f9e0b1c6 xml interface for blacklists
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2829 6c8d7289-2bf4-0310-a012-ef5d649a1542
2006-10-20 14:47:18 +00:00
allo
74f09a0510 some more xml-backend files.
ConfigAdvanced_p.java: list settings after changing.

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2784 6c8d7289-2bf4-0310-a012-ef5d649a1542
2006-10-15 21:41:47 +00:00
allo
e25172853a fixed license notice
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2714 6c8d7289-2bf4-0310-a012-ef5d649a1542
2006-10-07 22:25:05 +00:00
allo
1d0c0edda3 first version of posts/get from the del.icio.us api
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2713 6c8d7289-2bf4-0310-a012-ef5d649a1542
2006-10-07 22:16:09 +00:00
orbiter
5a40ea7866 refactoring of wget string list generation
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2692 6c8d7289-2bf4-0310-a012-ef5d649a1542
2006-10-02 09:59:20 +00:00
orbiter
dbc2e039bb added time-out option parameter to call hierarchy
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2691 6c8d7289-2bf4-0310-a012-ef5d649a1542
2006-10-02 09:40:18 +00:00
orbiter
b59d4576af increased version number to emphasise that the snippet fix
_dramatically_ increased search speed

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2690 6c8d7289-2bf4-0310-a012-ef5d649a1542
2006-10-02 01:50:57 +00:00
orbiter
d4c239e4be - fixed problem in collection index with deletion of single url references
- added automatic deletion of not-found snippets after search

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2689 6c8d7289-2bf4-0310-a012-ef5d649a1542
2006-10-02 01:40:52 +00:00
orbiter
df1629b05a - code cleanup
- version 0.471
- moved surftipps to own web page


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2676 6c8d7289-2bf4-0310-a012-ef5d649a1542
2006-09-29 22:27:20 +00:00
orbiter
3aac5b26da - added automatic tag generation when a web page from the search results is added
- added new image 'B' in front of search results for bookmark generation
- added news generation when a public bookmark is added
- the '+' in front of search results has new meaning: positive rating for that result
- added news generation when a '+' is hit

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2613 6c8d7289-2bf4-0310-a012-ef5d649a1542
2006-09-18 00:37:02 +00:00
orbiter
5015e780c2 - simplified watchCrawler code
- changed display of watchCrawler slightly

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2594 6c8d7289-2bf4-0310-a012-ef5d649a1542
2006-09-15 13:54:10 +00:00
orbiter
c89d8142bb replaced old 'kCache' by a full-controlled cache
there are now two full-controlled caches for incoming indexes:
- dhtIn
- dhtOut
during indexing, all indexes that shall not be transported to remote peers
because they belong to the own peer are stored to dhtIn. It is furthermore
ensured that received indexes are not again transmitted to other peers
directly. They may, however be transmitted later if the network grows.

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2574 6c8d7289-2bf4-0310-a012-ef5d649a1542
2006-09-14 00:51:02 +00:00
theli
92e986bb91 *) adding missing return prop (requested by allo)
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2532 6c8d7289-2bf4-0310-a012-ef5d649a1542
2006-09-09 12:34:24 +00:00
allo
f0529fe53e update for ftp urls
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2531 6c8d7289-2bf4-0310-a012-ef5d649a1542
2006-09-09 12:28:28 +00:00
theli
413e6b9855 *) direct access to responseheaders of sbQueue.Entry removed to make it more http independent
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2489 6c8d7289-2bf4-0310-a012-ef5d649a1542
2006-09-04 15:56:49 +00:00
theli
eb9b138986 *) next step of restructuring for new crawlers
- conversion of the crawler pool into a keyed object pool
   - crawlers are now loaded based on the url protocol (of course works only for http now)

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2473 6c8d7289-2bf4-0310-a012-ef5d649a1542
2006-09-04 06:52:55 +00:00
theli
1395aae742 *) starting restructuring which is needed to add crawlers for additional protocols
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2472 6c8d7289-2bf4-0310-a012-ef5d649a1542
2006-09-04 06:09:20 +00:00
orbiter
7df572756a fist step+attempt so solve the snippet marking problem.
See: http://www.yacy-forum.de/viewtopic.php?p=22855#22855

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2469 6c8d7289-2bf4-0310-a012-ef5d649a1542
2006-09-03 23:22:23 +00:00
orbiter
3879a0ecd0 replaced java.net.URL usage by use of new class de.anomic.net.URL
This shall be seen as an experiment to exclude all cases where
there could be a DNS lookup during URL comparisment.

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2290 6c8d7289-2bf4-0310-a012-ef5d649a1542
2006-07-13 01:21:53 +00:00
allo
933a9e02ab fix for broken build
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2284 6c8d7289-2bf4-0310-a012-ef5d649a1542
2006-07-09 14:35:20 +00:00
allo
360056b30c fix ajax bug (no valid xml)
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2283 6c8d7289-2bf4-0310-a012-ef5d649a1542
2006-07-09 10:59:55 +00:00
orbiter
90d569d70f refactoring of index management:
url storage is part of index management; moved plasmaURL to indexURL

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@2122 6c8d7289-2bf4-0310-a012-ef5d649a1542
2006-05-19 23:50:55 +00:00
allo
44d72f06c4 more Caching
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1965 6c8d7289-2bf4-0310-a012-ef5d649a1542
2006-03-24 15:08:27 +00:00
allo
1a13c8b78e right wordCachesize after orbiters commit.
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1882 6c8d7289-2bf4-0310-a012-ef5d649a1542
2006-03-13 18:09:09 +00:00
allo
6b056610e3 updated watchcrawler for the recent changes
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1881 6c8d7289-2bf4-0310-a012-ef5d649a1542
2006-03-13 16:33:47 +00:00
orbiter
bcd99fe83e introduced a second RAM cache for DHT transfer
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@1880 6c8d7289-2bf4-0310-a012-ef5d649a1542
2006-03-13 10:43:12 +00:00