3266 Commits

Author	SHA1	Message	Date
lotus	c8451614f3	fix for overflow ... http://forum.yacy-websuche.de/viewtopic.php?p=11696#p11696 git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5440 6c8d7289-2bf4-0310-a012-ef5d649a1542	2009-01-05 18:28:27 +00:00
orbiter	c4c4c223b9	fixed a problem with attribute flags on RWI entries that prevented proper selection of index-of constraint ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5437 6c8d7289-2bf4-0310-a012-ef5d649a1542	2009-01-04 02:27:29 +00:00
orbiter	6072831235	no cr transmission for robinson peers ... see also: http://forum.yacy-websuche.de/viewtopic.php?p=10290#p10290 git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5436 6c8d7289-2bf4-0310-a012-ef5d649a1542	2009-01-03 23:44:42 +00:00
low012	afe98bc11c	*) added changes as proposed by Halborinda in http://forum.yacy-websuche.de/viewtopic.php?f=6&t=1674 ... *) changed indention git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5434 6c8d7289-2bf4-0310-a012-ef5d649a1542	2009-01-03 08:24:08 +00:00
orbiter	07fc115e90	removed active profiling in kelondroRowSet ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5433 6c8d7289-2bf4-0310-a012-ef5d649a1542	2009-01-02 12:33:06 +00:00
orbiter	be4c458951	refactoring (implemented Iterable in kelondroRowCollection) ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5432 6c8d7289-2bf4-0310-a012-ef5d649a1542	2009-01-02 11:38:20 +00:00
low012	bb5c2cd12e	*) ISINDEX parameters will not be put on commandline anymore to prevent possible security hazards (better safe than sorry). Parmeters will have to be read from QUERY_STRING in ISINDEX case too which does not seem to be uncommon behaviour for web servers: http://vms.pdv-systeme.de/users/martinv/cgi_basics/cgi_basics.html#Datenuebergabe ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5431 6c8d7289-2bf4-0310-a012-ef5d649a1542	2009-01-02 11:18:26 +00:00
orbiter	b6bba18c37	replaced the storing procedure for the index ram cache with a method that generates BLOBHeap-compatible dumps ... this is a migration step to support a new method to store the web index, which will also based on the same data structure. made also a lot of refactoring for a better structuring of the BLOBHeap class. git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5430 6c8d7289-2bf4-0310-a012-ef5d649a1542	2009-01-01 22:31:16 +00:00
low012	db1cfae3e7	*) cleaning up after myself ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5429 6c8d7289-2bf4-0310-a012-ef5d649a1542	2009-01-01 19:45:15 +00:00
low012	f547f9a78c	*) added CGI capabilities (run Perl scripts and other software via HTTP GET and POST) ... *) set cgi.allow to true in yacy.conf to enable CGI (CGI is disabled by default) *) edit cgi.suffixes in yacy.conf if necessary to use additional script types ATTENTION: This is a rather experimental feature, not all environment variables are set yet. Only enable CGI if you know what you are doing. Poorly implemented CGI scripts can put a system's integrity at risk! Implementation of more environment variables and documentation due for the next days. git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5428 6c8d7289-2bf4-0310-a012-ef5d649a1542	2009-01-01 19:40:06 +00:00
f1ori	bdc380cd84	* add lastModified to templateCache ... -> no outdated files from cache anymore... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5427 6c8d7289-2bf4-0310-a012-ef5d649a1542	2009-01-01 14:56:53 +00:00
f1ori	025094675f	* remove empty directory ... * add necessary dependency for pdfParser git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5424 6c8d7289-2bf4-0310-a012-ef5d649a1542	2008-12-31 19:39:02 +00:00
f1ori	c5691180cb	* skip style-tags in HTML-files ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5423 6c8d7289-2bf4-0310-a012-ef5d649a1542	2008-12-31 19:34:24 +00:00
orbiter	3567c58b18	added another filed information for BLOBHeap dumps: the gaps ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5420 6c8d7289-2bf4-0310-a012-ef5d649a1542	2008-12-30 10:49:43 +00:00
orbiter	abdd4aa414	added a index dump for blob heaps: ... this will increase the shutdown time for at most some seconds, but will speed up the start-up git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5419 6c8d7289-2bf4-0310-a012-ef5d649a1542	2008-12-29 21:36:27 +00:00
orbiter	8c3205b62e	fix for OOB Exception ... see http://forum.yacy-websuche.de/viewtopic.php?p=11598#p11598 git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5417 6c8d7289-2bf4-0310-a012-ef5d649a1542	2008-12-29 17:36:53 +00:00
orbiter	78c568331e	added test channel to /xml/feed.rss ... can be obtained with http://localhost:8080/xml/feed.rss?set=TEST returns always a single feed entry with a fresh date git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5416 6c8d7289-2bf4-0310-a012-ef5d649a1542	2008-12-29 12:39:07 +00:00
orbiter	e004da48d3	- added fast fingerprint computation for files (any). Will be used in new index dump method ... - refactoring git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5415 6c8d7289-2bf4-0310-a012-ef5d649a1542	2008-12-29 12:22:13 +00:00
f1ori	2d2ce24011	* remove all encoding-stuff from proxy ... encoding is handled by parsers or browser, proxy only passes through git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5410 6c8d7289-2bf4-0310-a012-ef5d649a1542	2008-12-23 19:14:54 +00:00
f1ori	73c8a0839c	* abort download, when proxy connection is closed ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5409 6c8d7289-2bf4-0310-a012-ef5d649a1542	2008-12-23 11:30:24 +00:00
orbiter	bb935fdbb0	less organization overhead for DNS caching and prefetching ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5408 6c8d7289-2bf4-0310-a012-ef5d649a1542	2008-12-23 10:06:49 +00:00
f1ori	4907697cfa	* make fileuploads through proxy bigger than 65500 bytes possible ... * remove gzip-encoding for files from cache git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5407 6c8d7289-2bf4-0310-a012-ef5d649a1542	2008-12-22 23:04:00 +00:00
orbiter	fc8189f3fb	better self-healing of corrupted databases ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5406 6c8d7289-2bf4-0310-a012-ef5d649a1542	2008-12-22 16:43:49 +00:00
f1ori	963da8c3f9	* updated tm-extractors to new version 1.0 ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5405 6c8d7289-2bf4-0310-a012-ef5d649a1542	2008-12-21 14:51:03 +00:00
orbiter	e34ac22fbd	- added new monitoring servlet at ... http://localhost:8080/PerformanceConcurrency_p.html - used the new monitoring to do some fine-tuning of the indexing queue git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5402 6c8d7289-2bf4-0310-a012-ef5d649a1542	2008-12-19 15:26:01 +00:00
lotus	449e697436	fix for null-seed in seedfile ... http://forum.yacy-websuche.de/viewtopic.php?f=5&t=1653 git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5401 6c8d7289-2bf4-0310-a012-ef5d649a1542	2008-12-19 12:10:01 +00:00
orbiter	d376d81fc4	replaced busy thread control of crawl stacker by blocking threads ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5400 6c8d7289-2bf4-0310-a012-ef5d649a1542	2008-12-18 23:18:34 +00:00
orbiter	f29b48d9ff	patch for IndexOutOfBoundsException ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5399 6c8d7289-2bf4-0310-a012-ef5d649a1542	2008-12-18 22:05:26 +00:00
f1ori	0881190b19	* Robots.txt: don't interpret Crawl-Delays for other robots ... fixes: http://forum.yacy-websuche.de/viewtopic.php?f=5&t=1647 git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5398 6c8d7289-2bf4-0310-a012-ef5d649a1542	2008-12-18 15:35:41 +00:00
orbiter	243e73f53b	removed unnecessary usage of kelondroBLOBTree ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5397 6c8d7289-2bf4-0310-a012-ef5d649a1542	2008-12-18 00:18:37 +00:00
orbiter	8cb7170b75	- set status of kelondroTree, kelondroBLOBTree and kelondroFlexTable to deprecated ... - removed initialization and/or usage of kelondroFlexTable (should meanwhile not be used any more) git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5396 6c8d7289-2bf4-0310-a012-ef5d649a1542	2008-12-18 00:08:17 +00:00
orbiter	7535fd7447	- refactoring of CrawlEntry and CrawlStacker ... - introduced blocking queues in CrawlStacker to make it ready for concurrency - added a second busy thread for the CrawlStacker The CrawlStacker is multithreaded. It shall be transformed into a BlockingThread in another step. The concurrency of the stacker will hopefully solve some problems with cases where DNS blocks. git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5395 6c8d7289-2bf4-0310-a012-ef5d649a1542	2008-12-17 22:53:06 +00:00
lotus	18513e2ee2	npe fix: http://forum.yacy-websuche.de/viewtopic.php?t=1646 ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5393 6c8d7289-2bf4-0310-a012-ef5d649a1542	2008-12-16 13:36:13 +00:00
orbiter	2802138787	- refactoring of CrawlStacker (to prepare it for new multi-Threading to remove DNS lookup bottleneck) ... - fix of shallBeOwnWord target computation heuristic git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5392 6c8d7289-2bf4-0310-a012-ef5d649a1542	2008-12-15 00:02:58 +00:00
orbiter	db6b3bf5a3	speed enhancement for integrated http server: ... - tuning hacks in template engine - bypassing the template engine if no servlet present git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5389 6c8d7289-2bf4-0310-a012-ef5d649a1542	2008-12-11 20:10:37 +00:00
orbiter	7cd08bd5fb	fix for NPE in BLOBCompressor ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5388 6c8d7289-2bf4-0310-a012-ef5d649a1542	2008-12-11 13:33:24 +00:00
orbiter	5b94498643	fine-tuning of cache usage from SVN 5386 and a bug fix for overflow in available() method ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5387 6c8d7289-2bf4-0310-a012-ef5d649a1542	2008-12-10 14:35:01 +00:00
orbiter	1779c3c507	- added a read cache to the RAFile interface to RandomAccessFile ... - added a write buffer to BLOBHeap - modified the BLOBBuffer (is now only to buffer non-compressed content) - added content compression to the HTCache The new read cache will decrease the start/initialization time of BLOB files, like the HTCache, RobotsTxt and other BLOBHeap structures. git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5386 6c8d7289-2bf4-0310-a012-ef5d649a1542	2008-12-10 11:15:19 +00:00
orbiter	e1acdb952c	fix for problem with userDB and bookmarksDB which was caused by changes in kelondroRA in SVN 5376 ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5385 6c8d7289-2bf4-0310-a012-ef5d649a1542	2008-12-08 00:17:45 +00:00
orbiter	4a2dac659e	more speed hacks: ... - modified and activated write buffer - increased cache flush factor - fixed a problem with deadlocking of indexing process git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5382 6c8d7289-2bf4-0310-a012-ef5d649a1542	2008-12-05 13:55:48 +00:00
orbiter	47292e696a	more performance hacks ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5379 6c8d7289-2bf4-0310-a012-ef5d649a1542	2008-12-04 12:54:16 +00:00
orbiter	759cef23dd	fix for bug in kelondroAbstractRA.readFully ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5378 6c8d7289-2bf4-0310-a012-ef5d649a1542	2008-12-03 23:32:07 +00:00
orbiter	d39d420b39	performance hacks ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5376 6c8d7289-2bf4-0310-a012-ef5d649a1542	2008-12-03 15:38:29 +00:00
orbiter	0b4808ba3d	added new interactive search feature: ... - during the user types search queries, the local database is searched - results are presented interactively This was implemented using a new JSON result format for search results in YaCy - added JSON as file format for servlets - refactoring of current search servlets (xml and html) - added JSON output format for search results - added AJAX-based search page, that uses the yacysearch.json selrvlet to print results as a query is typed git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5373 6c8d7289-2bf4-0310-a012-ef5d649a1542	2008-12-02 15:24:25 +00:00
orbiter	74a3d86114	fixed a error response that might present classified information ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5372 6c8d7289-2bf4-0310-a012-ef5d649a1542	2008-12-01 23:14:42 +00:00
orbiter	c6525ab75f	fix for NPE in seed handling ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5371 6c8d7289-2bf4-0310-a012-ef5d649a1542	2008-12-01 23:08:27 +00:00
lotus	1951d30a62	addendum to last commit ... handle words with length < 3 correctly git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5369 6c8d7289-2bf4-0310-a012-ef5d649a1542	2008-11-26 19:43:40 +00:00
lotus	325ba7bfb8	only query words with length > 2 ... this is not complete, yet git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5368 6c8d7289-2bf4-0310-a012-ef5d649a1542	2008-11-26 16:41:38 +00:00
low012	e423fa9846	*) added method to only get file names in directory listing which match a filter ... *) only files which end with .black will be listed as blacklists *) added a little bit of Javadoc git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5366 6c8d7289-2bf4-0310-a012-ef5d649a1542	2008-11-25 20:26:06 +00:00
orbiter	513179f404	changed interface to colletctionIndex and adopted all implementing classes: ... do not return a result of a double-check when adding entries with addUnique git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5363 6c8d7289-2bf4-0310-a012-ef5d649a1542	2008-11-23 23:55:08 +00:00