Marown/yacy_search_server
1
0
Fork 0
mirror of https://github.com/yacy/yacy_search_server.git synced 2024-09-19 00:01:41 +02:00
Code Issues Actions Packages Projects Releases Wiki Activity
6,158 Commits 1 Branch 20 Tags 836 MiB
94b2a664f3
Commit Graph

6158 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
orbiter
94b2a664f3 - use a static DiskFileItemFactory (one instantiation is enough) 
- use more memory for the DiskFileItemFactory to avoid IO when POST commands come

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6485 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-11-18 15:05:51 +00:00
orbiter
267108470f testing jmx console for yacy: 
- start YaCy with startYACY.sh -l
- open jconsole and open localhost:9999

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6484 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-11-18 14:54:30 +00:00
suessthomas
d9835d8568 Conversion of the de. Language to UTF-8 format. This version will replace the now used language File in the future. Please test and report bugs to me (th-suess@gmx.de). 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6483 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-11-17 16:19:46 +00:00
orbiter
fd0658ce7c avoid forced execution of InetAddress.getLocalHost() at startup, because that hangs at some strangely declared linux configurations. The Domains.localHostAddresses object is first instantiated with a more simple logic and enriched with more host addresses using a concurrent thread that will not block a startup process. 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6482 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-11-16 23:08:20 +00:00
orbiter
013f337d3f - avoid unnecessary host name lookups for localhost 
- avoid unnecessary reverse domain name lookups for remote access

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6481 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-11-16 23:00:54 +00:00
low012
141712ec95 *) small changes to UI 
*) password will not be deleted anymore when changing to unlimited access from localhost

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6480 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-11-15 22:15:25 +00:00
lotus
12dd8ece3e enabled memory protection from 6459 with 50000kb (disables dht-in) 
this should only apply if there is really little memory available because it is checked by threads explictly requesting memory

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6479 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-11-13 16:26:45 +00:00
orbiter
20c5d78a5c fix for a ConcurrentModificationException 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6478 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-11-11 23:31:12 +00:00
orbiter
5afd9f7a91 fix for crlf writing 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6477 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-11-11 22:50:44 +00:00
orbiter
7144d2df6e added crawlReceipt servlet as individual class to examine OOM problem as documented in 
http://forum.yacy-websuche.de/viewtopic.php?p=18120#p18120

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6476 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-11-11 16:12:00 +00:00
orbiter
2d3c98b742 less computation within synchronized blocks 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6475 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-11-11 16:07:40 +00:00
orbiter
1a146b0d73 added a patch to ignore bad mime-ignore patterns 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6474 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-11-11 15:49:53 +00:00
orbiter
29fe436e36 - fixed post-ranking including prefer mask 
- enhanced a core database access method / less wasted ram

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6473 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-11-09 19:14:51 +00:00
orbiter
e9ab130ad7 fixed start/stop using ant 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6472 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-11-09 19:12:33 +00:00
orbiter
5399d1e2bc refactoring (reason: get more abstraction to use the blacklist class; for integration in other servlets) 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6471 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-11-08 22:58:57 +00:00
orbiter
a97fdb4566 catch for NPE in image parser 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6470 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-11-07 23:39:31 +00:00
low012
9ee7862710 *) added configuration script 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6469 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-11-07 13:21:27 +00:00
orbiter
534182559c removed concurrency hacks from SplitTable because it showed deadlock-like situation. 
see thread dump at http://forum.yacy-websuche.de/viewtopic.php?p=18081#p18081

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6468 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-11-07 11:52:03 +00:00
orbiter
1fa0ac26e9 better protection against NPEs during search/ranking 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6467 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-11-07 10:58:33 +00:00
orbiter
2bab0679e0 lost my key :-( 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6466 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-11-06 23:46:29 +00:00
orbiter
4c99d4683d possible fix for lost crawl profile handles: clean-up job did wrong measurement to see if crawl is still running. 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6465 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-11-06 23:15:20 +00:00
orbiter
cd6745b292 accept rss feeds without channel descriptions 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6464 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-11-06 22:46:21 +00:00
orbiter
08f1cbb125 another update to the pdf parser 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6463 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-11-06 22:41:37 +00:00
orbiter
54c54fb144 get a handle for grep: 'StackTrace' 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6462 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-11-06 19:55:21 +00:00
orbiter
605e896d6c more details for exception catching when parsing pdfs 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6461 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-11-06 19:47:24 +00:00
orbiter
18b21eaffe small fixes to search default values and server logging 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6460 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-11-06 19:13:35 +00:00
lotus
6edc168cfe option to disable dht by memory limit: 
memory.acceptDHT in kbytes
not yet pre-enabled, will clear on every startup
please review since this could break dht in freeworld

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6459 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-11-06 19:13:30 +00:00
orbiter
4431b9767e added about 450 replacements for printStackTrace() methods to pipe such traces into the log at DATA/LOG/ 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6458 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-11-05 20:28:37 +00:00
orbiter
e3025ee691 - new icon for OAI-PMH loading action 
- added many stack trace outputs for exceptions in crawl profile handler to find the 'missing profile handle' bug
- catched one more timeout exception in httpd file loader

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6457 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-11-05 16:40:15 +00:00
orbiter
f0b8db93f0 - more abstraction of serverCore thread access 
- no more keep-alive when number of connections exceeds 1/2 of the allowed number of connection

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6456 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-11-05 14:54:43 +00:00
orbiter
19f31bb043 - moved OAI-PMH source list file from SETTINGS to DICTIONARIES/harvesting 
- added convenience method for loading of files from the web in LoaderDispatcher

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6455 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-11-04 16:15:28 +00:00
orbiter
2889b9426e missing code for last commit 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6454 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-11-04 12:03:19 +00:00
orbiter
b6a8887ff5 better handling of running sessions without explicit hashtable 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6453 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-11-04 11:59:15 +00:00
orbiter
1dc7ea986a added a dynamic keep-alive time-out for http server sessions: 
if there are many concurrent server sessions, the timout is decreased.
This should avoid a situation where the clean-up thread is too
late to stop running http sessions that should be terminated
before the maximum number of server sessions is reached.

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6452 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-11-04 11:01:09 +00:00
low012
e77c906673 *) minor changes mainly in comments 
*) added svn:keyword settings for several files

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6451 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-11-03 22:47:53 +00:00
low012
f1740edbf8 *) added skript to change memory settings, password and port (experimental, don't blame me if it messes up your configuration) 
*) minor change in Digest class, added option in main method, might not be optimal yet

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6450 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-11-03 22:28:29 +00:00
orbiter
11f7da06ed - fixes to csv parser 
- automatic OAI-PMH import by just clicking on one link from the provided resource list

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6449 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-11-03 21:18:19 +00:00
orbiter
9b6762ec2e - added a csv "comma separated values" parser to parse OAI-PMH sources from 
http://roar.eprints.org/index.php?action=csv
- integrated the csv parser into the crawlers parser list
- added an extension to the OAI-PMH import function to download and show the roar csv file using the csv parser

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6448 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-11-03 20:10:59 +00:00
orbiter
0f63de8236 - it is now possible to start several OAI imports concurrently 
(still not possible to start them with one single request, that will be next)
- added a monitor for all running and finished OAI imports (with a little bit of animation..)



git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6447 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-11-03 16:15:22 +00:00
orbiter
176e334aa4 fixes 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6446 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-11-02 19:23:05 +00:00
orbiter
2fa6bf440b workflow update to OAI-PMH importer 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6445 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-11-02 18:19:30 +00:00
orbiter
b0b7a4f9a5 - added function to OAI-PMH reader that can pull all records from a server using an evaluation of the resumption token to get URL to retrieve remaining records 
- added monitoring for retrieved records

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6444 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-11-02 11:53:14 +00:00
orbiter
350d13e153 very first working version of oai-pmh importer: if given the right url, the importer can read and index listRecord xml files and calculate the right resumptionURL which is then given as next default start point for the importer url input. 
no automatic harvesting by now, this will be done later

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6443 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-11-02 00:14:14 +00:00
lotus
58616d99e4 patch for yacy disk usage detection on lvm host 
by Michael S.

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6442 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-11-01 08:54:16 +00:00
lotus
79251e6f60 configurable disk space hardlimit for dht 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6441 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-10-31 19:12:53 +00:00
orbiter
a0e891c63d - some redesign in UI menu structure to make room for new 'Content Integration' main menu containing import servlets for Wikimedia Dumps, phpbb3 forum imports and OAI-PMH imports 
- extended the OAI-PMH test applet and integrated it into the menu. Does still not import OAI-PMH records, but shows that it is able to read and parse this data
- some redesign in ZURL storage: refactoring of access methods, better concurrency, less synchronization
- added a limitation to the LURL metadata database table cache to 20 million entries: this cache was until now not limited and only limited by the available RAM which may have caused a memory-leak-like behavior.

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6440 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-10-31 11:58:06 +00:00
orbiter
8a1046feaa less maximum file size, too many problems with larger size 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6439 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-10-30 20:21:45 +00:00
orbiter
4240785f20 added anti-alias function for line drawing 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6438 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-10-30 15:58:36 +00:00
orbiter
30f108f97d added stub of oai-pmh importer (not working yet) 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6437 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-10-30 15:58:04 +00:00
orbiter
77c99e500f added more control over memory allocation 
should avoid some of the OOMs

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6436 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-10-27 15:25:48 +00:00