65 Commits

Author	SHA1	Message	Date
orbiter	491ba6a1ba	- some refactoring in workflow ... - some refactoring in search process - fixed image search for json and rss output - search navigation on bottom of search result page in cases where there are more than 6 results on page - fixes for number of displayed documents - disabled pseudostemming git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6504 6c8d7289-2bf4-0310-a012-ef5d649a1542	2009-11-24 11:13:11 +00:00
orbiter	969123385b	added json and rss output for image search ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6503 6c8d7289-2bf4-0310-a012-ef5d649a1542	2009-11-23 16:10:50 +00:00
orbiter	d183f8d980	refactoring (moved code from ContentTransformer to TemplateEngine) ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6498 6c8d7289-2bf4-0310-a012-ef5d649a1542	2009-11-20 14:57:00 +00:00
orbiter	23aef43786	- better synchronization in SortStack ... - better ThreadGroup organization - less worker threads for media search (64 was too much...) git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6497 6c8d7289-2bf4-0310-a012-ef5d649a1542	2009-11-20 14:35:33 +00:00
orbiter	7b1f5b0430	- better media search ranking ... - better concurrency with enhanced synchronization in sort stack git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6496 6c8d7289-2bf4-0310-a012-ef5d649a1542	2009-11-20 13:19:12 +00:00
orbiter	4df88a4e7a	- fixes for missing or bad hashCode computation ... - fixes for bad equals() methods that had not been used by hash maps and therefore some classes did not work as objects in hash maps. - this may also affect some cases where double-checks should have been, but did not work. git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6495 6c8d7289-2bf4-0310-a012-ef5d649a1542	2009-11-20 12:11:56 +00:00
orbiter	dbdf2570ba	added comparator and more fixes for SortStack/SortStore ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6494 6c8d7289-2bf4-0310-a012-ef5d649a1542	2009-11-20 03:30:48 +00:00
orbiter	d2938c44a1	- added bmp parser to the document parsers ... - image parser that implement the document parser interface return itself in the list of images of the document which should cause that the parsed images contribute to the image search git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6493 6c8d7289-2bf4-0310-a012-ef5d649a1542	2009-11-19 23:22:53 +00:00
orbiter	1dff620181	Better implementation of SortStack and SortStore and adoptions in all using classes to implement the necessary Comparable interface and hash code computation. ... The better SortStack performance affects crawling and image search speed and quality. git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6492 6c8d7289-2bf4-0310-a012-ef5d649a1542	2009-11-19 13:49:28 +00:00
orbiter	fe41a84330	some enhancements in web caching: avoid double loading of response metadata and/or content ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6491 6c8d7289-2bf4-0310-a012-ef5d649a1542	2009-11-19 10:17:26 +00:00
orbiter	06d0dcde20	more enhancements to image search ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6490 6c8d7289-2bf4-0310-a012-ef5d649a1542	2009-11-19 00:43:42 +00:00
orbiter	4c6312d103	enhanced image search ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6489 6c8d7289-2bf4-0310-a012-ef5d649a1542	2009-11-18 23:56:05 +00:00
orbiter	2d8f3ee301	some performance hacks ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6488 6c8d7289-2bf4-0310-a012-ef5d649a1542	2009-11-18 16:03:28 +00:00
orbiter	fd0658ce7c	avoid forced execution of InetAddress.getLocalHost() at startup, because that hangs at some strangely declared linux configurations. The Domains.localHostAddresses object is first instantiated with a more simple logic and enriched with more host addresses using a concurrent thread that will not block a startup process. ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6482 6c8d7289-2bf4-0310-a012-ef5d649a1542	2009-11-16 23:08:20 +00:00
orbiter	013f337d3f	- avoid unnecessary host name lookups for localhost ... - avoid unnecessary reverse domain name lookups for remote access git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6481 6c8d7289-2bf4-0310-a012-ef5d649a1542	2009-11-16 23:00:54 +00:00
orbiter	5afd9f7a91	fix for crlf writing ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6477 6c8d7289-2bf4-0310-a012-ef5d649a1542	2009-11-11 22:50:44 +00:00
orbiter	2d3c98b742	less computation within synchronized blocks ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6475 6c8d7289-2bf4-0310-a012-ef5d649a1542	2009-11-11 16:07:40 +00:00
orbiter	1a146b0d73	added a patch to ignore bad mime-ignore patterns ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6474 6c8d7289-2bf4-0310-a012-ef5d649a1542	2009-11-11 15:49:53 +00:00
orbiter	29fe436e36	- fixed post-ranking including prefer mask ... - enhanced a core database access method / less wasted ram git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6473 6c8d7289-2bf4-0310-a012-ef5d649a1542	2009-11-09 19:14:51 +00:00
orbiter	5399d1e2bc	refactoring (reason: get more abstraction to use the blacklist class; for integration in other servlets) ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6471 6c8d7289-2bf4-0310-a012-ef5d649a1542	2009-11-08 22:58:57 +00:00
orbiter	a97fdb4566	catch for NPE in image parser ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6470 6c8d7289-2bf4-0310-a012-ef5d649a1542	2009-11-07 23:39:31 +00:00
orbiter	534182559c	removed concurrency hacks from SplitTable because it showed deadlock-like situation. ... see thread dump at http://forum.yacy-websuche.de/viewtopic.php?p=18081#p18081 git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6468 6c8d7289-2bf4-0310-a012-ef5d649a1542	2009-11-07 11:52:03 +00:00
orbiter	cd6745b292	accept rss feeds without channel descriptions ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6464 6c8d7289-2bf4-0310-a012-ef5d649a1542	2009-11-06 22:46:21 +00:00
orbiter	08f1cbb125	another update to the pdf parser ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6463 6c8d7289-2bf4-0310-a012-ef5d649a1542	2009-11-06 22:41:37 +00:00
orbiter	54c54fb144	get a handle for grep: 'StackTrace' ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6462 6c8d7289-2bf4-0310-a012-ef5d649a1542	2009-11-06 19:55:21 +00:00
orbiter	605e896d6c	more details for exception catching when parsing pdfs ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6461 6c8d7289-2bf4-0310-a012-ef5d649a1542	2009-11-06 19:47:24 +00:00
lotus	6edc168cfe	option to disable dht by memory limit: ... memory.acceptDHT in kbytes not yet pre-enabled, will clear on every startup please review since this could break dht in freeworld git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6459 6c8d7289-2bf4-0310-a012-ef5d649a1542	2009-11-06 19:13:30 +00:00
orbiter	4431b9767e	added about 450 replacements for printStackTrace() methods to pipe such traces into the log at DATA/LOG/ ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6458 6c8d7289-2bf4-0310-a012-ef5d649a1542	2009-11-05 20:28:37 +00:00
orbiter	19f31bb043	- moved OAI-PMH source list file from SETTINGS to DICTIONARIES/harvesting ... - added convenience method for loading of files from the web in LoaderDispatcher git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6455 6c8d7289-2bf4-0310-a012-ef5d649a1542	2009-11-04 16:15:28 +00:00
low012	e77c906673	*) minor changes mainly in comments ... *) added svn:keyword settings for several files git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6451 6c8d7289-2bf4-0310-a012-ef5d649a1542	2009-11-03 22:47:53 +00:00
low012	f1740edbf8	*) added skript to change memory settings, password and port (experimental, don't blame me if it messes up your configuration) ... *) minor change in Digest class, added option in main method, might not be optimal yet git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6450 6c8d7289-2bf4-0310-a012-ef5d649a1542	2009-11-03 22:28:29 +00:00
orbiter	11f7da06ed	- fixes to csv parser ... - automatic OAI-PMH import by just clicking on one link from the provided resource list git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6449 6c8d7289-2bf4-0310-a012-ef5d649a1542	2009-11-03 21:18:19 +00:00
orbiter	9b6762ec2e	- added a csv "comma separated values" parser to parse OAI-PMH sources from ... http://roar.eprints.org/index.php?action=csv - integrated the csv parser into the crawlers parser list - added an extension to the OAI-PMH import function to download and show the roar csv file using the csv parser git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6448 6c8d7289-2bf4-0310-a012-ef5d649a1542	2009-11-03 20:10:59 +00:00
orbiter	176e334aa4	fixes ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6446 6c8d7289-2bf4-0310-a012-ef5d649a1542	2009-11-02 19:23:05 +00:00
orbiter	2fa6bf440b	workflow update to OAI-PMH importer ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6445 6c8d7289-2bf4-0310-a012-ef5d649a1542	2009-11-02 18:19:30 +00:00
orbiter	b0b7a4f9a5	- added function to OAI-PMH reader that can pull all records from a server using an evaluation of the resumption token to get URL to retrieve remaining records ... - added monitoring for retrieved records git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6444 6c8d7289-2bf4-0310-a012-ef5d649a1542	2009-11-02 11:53:14 +00:00
orbiter	350d13e153	very first working version of oai-pmh importer: if given the right url, the importer can read and index listRecord xml files and calculate the right resumptionURL which is then given as next default start point for the importer url input. ... no automatic harvesting by now, this will be done later git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6443 6c8d7289-2bf4-0310-a012-ef5d649a1542	2009-11-02 00:14:14 +00:00
lotus	58616d99e4	patch for yacy disk usage detection on lvm host ... by Michael S. git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6442 6c8d7289-2bf4-0310-a012-ef5d649a1542	2009-11-01 08:54:16 +00:00
orbiter	a0e891c63d	- some redesign in UI menu structure to make room for new 'Content Integration' main menu containing import servlets for Wikimedia Dumps, phpbb3 forum imports and OAI-PMH imports ... - extended the OAI-PMH test applet and integrated it into the menu. Does still not import OAI-PMH records, but shows that it is able to read and parse this data - some redesign in ZURL storage: refactoring of access methods, better concurrency, less synchronization - added a limitation to the LURL metadata database table cache to 20 million entries: this cache was until now not limited and only limited by the available RAM which may have caused a memory-leak-like behavior. git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6440 6c8d7289-2bf4-0310-a012-ef5d649a1542	2009-10-31 11:58:06 +00:00
orbiter	4240785f20	added anti-alias function for line drawing ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6438 6c8d7289-2bf4-0310-a012-ef5d649a1542	2009-10-30 15:58:36 +00:00
orbiter	30f108f97d	added stub of oai-pmh importer (not working yet) ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6437 6c8d7289-2bf4-0310-a012-ef5d649a1542	2009-10-30 15:58:04 +00:00
orbiter	77c99e500f	added more control over memory allocation ... should avoid some of the OOMs git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6436 6c8d7289-2bf4-0310-a012-ef5d649a1542	2009-10-27 15:25:48 +00:00
orbiter	52470d0de4	- fix for xls parser ... - fix for image parser - temporary integration of images as document types in the crawler and indexer for testing of the image parser git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6435 6c8d7289-2bf4-0310-a012-ef5d649a1542	2009-10-22 22:38:04 +00:00
orbiter	5e8038ac4d	- refactoring of blacklists ... - refactoring of event origin encoding git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6434 6c8d7289-2bf4-0310-a012-ef5d649a1542	2009-10-21 20:14:30 +00:00
orbiter	26fafd85a5	- more refactoring ... - fixed problem with parsers git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6433 6c8d7289-2bf4-0310-a012-ef5d649a1542	2009-10-21 15:12:34 +00:00
orbiter	3528b970d6	- refactoring ... - added new experimental (not-yet-working) image parser - added new test image git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6431 6c8d7289-2bf4-0310-a012-ef5d649a1542	2009-10-19 22:34:44 +00:00
orbiter	a8ce192f63	- shifted main classes to new package net.yacy ... - fixed some bugs in last commit git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6427 6c8d7289-2bf4-0310-a012-ef5d649a1542	2009-10-18 01:38:07 +00:00
orbiter	b79f4f062f	refactoring of yacy documents and parsers: they depend now only on the kelondro classes ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6426 6c8d7289-2bf4-0310-a012-ef5d649a1542	2009-10-18 00:53:43 +00:00
orbiter	c864901087	- moved httpd.mime to defaults path ... - some documentation fixes - adopted a default setting for the search window: moves css setting to base.css - some enhancements for the DocumentIndex class git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6410 6c8d7289-2bf4-0310-a012-ef5d649a1542	2009-10-14 13:29:09 +00:00
orbiter	6192205533	more final modifier ... git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6407 6c8d7289-2bf4-0310-a012-ef5d649a1542	2009-10-12 21:59:39 +00:00