Marown/yacy_search_server
1
0
Fork 0
mirror of https://github.com/yacy/yacy_search_server.git synced 2024-09-21 00:00:13 +02:00
Code Issues Actions Packages Projects Releases Wiki Activity
6,227 Commits 1 Branch 20 Tags 836 MiB
32972139af
Commit Graph

3973 Commits

Author SHA1 Message Date
orbiter
bb2e03761c - fix for deadlock with 100% CPU during search 
- fix for failure of ranking because of a ConcurrentModificationException

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6553 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2010-01-07 12:41:43 +00:00
orbiter
dff4f95c78 some patches to get the torrent parser working 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6551 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2010-01-07 00:42:12 +00:00
low012
82198acc06 *) minor changes 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6537 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-12-28 11:06:49 +00:00
low012
b75547fc60 *) minor changes 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6536 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-12-28 11:05:50 +00:00
orbiter
57d729e377 fix for negative numbers in network statistic 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6532 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-12-21 11:36:48 +00:00
orbiter
4ac4fe952c patch for npe in bookmarks 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6530 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-12-18 10:14:05 +00:00
orbiter
d548bd41ad fix for a npe during search 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6528 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-12-15 12:15:39 +00:00
orbiter
37245430c3 fix for NPE during DHT RWI selection 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6527 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-12-15 00:02:10 +00:00
orbiter
a37878b7d5 url parser regex performance hack 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6524 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-12-10 14:40:32 +00:00
orbiter
b527d2ebfa fix for media search 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6522 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-12-09 23:47:37 +00:00
orbiter
362b7a929b added extensive memory protection logic to avoid out of memory errors that may be caused by the RowCollection memory allocation function 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6521 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-12-09 23:27:26 +00:00
orbiter
8281e29963 - more configuration for profiling graph (number of events) 
- more logging for a shutdown: print reason and accessing IP into log


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6520 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-12-08 14:25:51 +00:00
f1ori
5f0f6b71b4 * revert last commit, something is more broken than before 
* UTC timestamps and lastseen-properteries still needs some debugging


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6519 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-12-06 21:54:32 +00:00
f1ori
8c8b642eba * fix timezone problem 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6518 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-12-06 21:03:12 +00:00
orbiter
4782d2c438 fix for search bug that appeared when looking at page 3 of results or further 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6515 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-12-03 12:25:03 +00:00
orbiter
29fde9ed49 better control of ranking order in sort stack 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6514 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-12-03 00:36:07 +00:00
orbiter
66923ebc6c - modified method in RequestHeader that delivers the host name of requester: no more reverse domain lookup (may have killed interface performance in some cases) 
- added logging output for shutdown servlet: show ip of requester of the shutdown

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6512 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-12-02 22:56:28 +00:00
orbiter
e34e63a039 preset of proper HashMap dimensions: should prevent re-hashing and increase performance 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6511 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-12-02 14:01:19 +00:00
orbiter
4a5100789f replaced _all_ size() == 0 with isEmpty() and all size() > 0 with !isEmpty(). The isEmpty() method is much faster in some cases, especially when used to access badly balanced hashtables where an size() operation becomes a large iteration. 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6510 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-12-02 00:37:59 +00:00
orbiter
f4946eaf27 - better thread dump 
- suppressed one server exception

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6509 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-12-01 22:53:36 +00:00
orbiter
9743b70d1c disabled keep-alive of server, not really needed for speed but a cause for much trouble and memory occupancy 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6508 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-12-01 19:14:16 +00:00
orbiter
491ba6a1ba - some refactoring in workflow 
- some refactoring in search process
- fixed image search for json and rss output
- search navigation on bottom of search result page in cases where there are more than 6 results on page
- fixes for number of displayed documents
- disabled pseudostemming

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6504 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-11-24 11:13:11 +00:00
orbiter
969123385b added json and rss output for image search 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6503 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-11-23 16:10:50 +00:00
orbiter
d183f8d980 refactoring (moved code from ContentTransformer to TemplateEngine) 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6498 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-11-20 14:57:00 +00:00
orbiter
23aef43786 - better synchronization in SortStack 
- better ThreadGroup organization
- less worker threads for media search (64 was too much...)


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6497 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-11-20 14:35:33 +00:00
orbiter
7b1f5b0430 - better media search ranking 
- better concurrency with enhanced synchronization in sort stack

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6496 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-11-20 13:19:12 +00:00
orbiter
4df88a4e7a - fixes for missing or bad hashCode computation 
- fixes for bad equals() methods that had not been used by hash maps and therefore some classes did not work as objects in hash maps.
- this may also affect some cases where double-checks should have been, but did not work.

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6495 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-11-20 12:11:56 +00:00
orbiter
dbdf2570ba added comparator and more fixes for SortStack/SortStore 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6494 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-11-20 03:30:48 +00:00
orbiter
1dff620181 Better implementation of SortStack and SortStore and adoptions in all using classes to implement the necessary Comparable interface and hash code computation. 
The better SortStack performance affects crawling and image search speed and quality.

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6492 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-11-19 13:49:28 +00:00
orbiter
fe41a84330 some enhancements in web caching: avoid double loading of response metadata and/or content 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6491 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-11-19 10:17:26 +00:00
orbiter
06d0dcde20 more enhancements to image search 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6490 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-11-19 00:43:42 +00:00
orbiter
4c6312d103 enhanced image search 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6489 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-11-18 23:56:05 +00:00
orbiter
2d8f3ee301 some performance hacks 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6488 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-11-18 16:03:28 +00:00
orbiter
94b2a664f3 - use a static DiskFileItemFactory (one instantiation is enough) 
- use more memory for the DiskFileItemFactory to avoid IO when POST commands come

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6485 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-11-18 15:05:51 +00:00
orbiter
013f337d3f - avoid unnecessary host name lookups for localhost 
- avoid unnecessary reverse domain name lookups for remote access

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6481 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-11-16 23:00:54 +00:00
orbiter
20c5d78a5c fix for a ConcurrentModificationException 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6478 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-11-11 23:31:12 +00:00
orbiter
7144d2df6e added crawlReceipt servlet as individual class to examine OOM problem as documented in 
http://forum.yacy-websuche.de/viewtopic.php?p=18120#p18120

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6476 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-11-11 16:12:00 +00:00
orbiter
29fe436e36 - fixed post-ranking including prefer mask 
- enhanced a core database access method / less wasted ram

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6473 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-11-09 19:14:51 +00:00
orbiter
5399d1e2bc refactoring (reason: get more abstraction to use the blacklist class; for integration in other servlets) 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6471 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-11-08 22:58:57 +00:00
orbiter
1fa0ac26e9 better protection against NPEs during search/ranking 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6467 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-11-07 10:58:33 +00:00
orbiter
4c99d4683d possible fix for lost crawl profile handles: clean-up job did wrong measurement to see if crawl is still running. 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6465 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-11-06 23:15:20 +00:00
orbiter
18b21eaffe small fixes to search default values and server logging 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6460 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-11-06 19:13:35 +00:00
lotus
6edc168cfe option to disable dht by memory limit: 
memory.acceptDHT in kbytes
not yet pre-enabled, will clear on every startup
please review since this could break dht in freeworld

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6459 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-11-06 19:13:30 +00:00
orbiter
4431b9767e added about 450 replacements for printStackTrace() methods to pipe such traces into the log at DATA/LOG/ 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6458 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-11-05 20:28:37 +00:00
orbiter
e3025ee691 - new icon for OAI-PMH loading action 
- added many stack trace outputs for exceptions in crawl profile handler to find the 'missing profile handle' bug
- catched one more timeout exception in httpd file loader

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6457 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-11-05 16:40:15 +00:00
orbiter
f0b8db93f0 - more abstraction of serverCore thread access 
- no more keep-alive when number of connections exceeds 1/2 of the allowed number of connection

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6456 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-11-05 14:54:43 +00:00
orbiter
2889b9426e missing code for last commit 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6454 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-11-04 12:03:19 +00:00
orbiter
b6a8887ff5 better handling of running sessions without explicit hashtable 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6453 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-11-04 11:59:15 +00:00
orbiter
1dc7ea986a added a dynamic keep-alive time-out for http server sessions: 
if there are many concurrent server sessions, the timout is decreased.
This should avoid a situation where the clean-up thread is too
late to stop running http sessions that should be terminated
before the maximum number of server sessions is reached.

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6452 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-11-04 11:01:09 +00:00
orbiter
b0b7a4f9a5 - added function to OAI-PMH reader that can pull all records from a server using an evaluation of the resumption token to get URL to retrieve remaining records 
- added monitoring for retrieved records

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6444 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-11-02 11:53:14 +00:00