Marown/yacy_search_server
1
0
Fork 0
mirror of https://github.com/yacy/yacy_search_server.git synced 2024-09-21 00:00:13 +02:00
Code Issues Actions Packages Projects Releases Wiki Activity
6,244 Commits 1 Branch 20 Tags 836 MiB
eb79ceb3ff
Commit Graph

3986 Commits

Author SHA1 Message Date
orbiter
18172451a0 better search computation: 
- increased sort limit, now 3000 entries, before: 1000
  this should cause that more results can be shown in case
  of strong limitating constraints, like domain navigation
- enhanced the sort process
- check against domain navigator bugs
- fix in sort stack
- showing now all naviagtion pages at first search (not only next page)


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6569 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2010-01-12 15:01:44 +00:00
orbiter
d126d6c1b5 renamed the servlet WatchCrawler_p to Crawler_p 
this was done because that servlet may be used for wget/cronjob
triggered crawl starts and it appears to be confusing that the
name of the crawl start servlet looks like a pure monitoring tool.


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6568 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2010-01-12 10:05:28 +00:00
orbiter
66c0a8e849 more PMD recommendations 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6567 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2010-01-11 22:18:38 +00:00
orbiter
909a4f91c7 added a logging output for crawl starts that shows the URL that can be used to start the crawl again 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6566 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2010-01-11 18:10:39 +00:00
orbiter
bc96d74813 - clean-up of robots.txt parser 
- added 'yacybot' as key to recognize robots.txt entries for YaCy
- removed unused method to get robots.txt from database

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6565 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2010-01-11 16:36:30 +00:00
orbiter
dd459281c8 applied code changes that are recommended by PMD 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6563 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2010-01-10 23:09:48 +00:00
lotus
eac2daf2e8 * reenable DHT if yet enough memory is available 
* reset treshold on reconfiguratoin
(thanks to sixcooler)

* display status message in web interface

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6562 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2010-01-10 19:04:43 +00:00
orbiter
d77a8f3b3e added some modifications recommended by PMD for better performance 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6560 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2010-01-10 01:40:26 +00:00
orbiter
d1973bae2a code cleanup: removed unused code and unused methods 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6559 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2010-01-10 00:42:49 +00:00
orbiter
a3b8b7b5c5 some redesign of the main menu structure: 
- moved all index generation servlets to it's own main menu item, including proxy indexing
- removed external index import because this operation is not recommended any more. Joining an index can simply be done by moving the index files from one peer to the other peer; they will be merged automatically
- fix to prevent endless loops when disconnecting http sessions
- fix to prevent application of bad blacklist entries that can cause a 'Dangling meta character' exception

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6558 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2010-01-10 00:10:43 +00:00
lotus
ab3cf60dbe fix for npe 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6557 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2010-01-09 14:10:27 +00:00
orbiter
7f20963b41 add-on to last commit 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6556 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2010-01-09 00:17:39 +00:00
orbiter
eeca2ded92 fix for http://forum.yacy-websuche.de/viewtopic.php?p=18500#p18500 
- catch uncatched OOM
- less wasting of memory

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6555 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2010-01-09 00:08:16 +00:00
orbiter
bb2e03761c - fix for deadlock with 100% CPU during search 
- fix for failure of ranking because of a ConcurrentModificationException

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6553 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2010-01-07 12:41:43 +00:00
orbiter
dff4f95c78 some patches to get the torrent parser working 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6551 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2010-01-07 00:42:12 +00:00
low012
82198acc06 *) minor changes 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6537 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-12-28 11:06:49 +00:00
low012
b75547fc60 *) minor changes 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6536 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-12-28 11:05:50 +00:00
orbiter
57d729e377 fix for negative numbers in network statistic 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6532 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-12-21 11:36:48 +00:00
orbiter
4ac4fe952c patch for npe in bookmarks 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6530 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-12-18 10:14:05 +00:00
orbiter
d548bd41ad fix for a npe during search 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6528 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-12-15 12:15:39 +00:00
orbiter
37245430c3 fix for NPE during DHT RWI selection 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6527 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-12-15 00:02:10 +00:00
orbiter
a37878b7d5 url parser regex performance hack 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6524 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-12-10 14:40:32 +00:00
orbiter
b527d2ebfa fix for media search 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6522 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-12-09 23:47:37 +00:00
orbiter
362b7a929b added extensive memory protection logic to avoid out of memory errors that may be caused by the RowCollection memory allocation function 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6521 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-12-09 23:27:26 +00:00
orbiter
8281e29963 - more configuration for profiling graph (number of events) 
- more logging for a shutdown: print reason and accessing IP into log


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6520 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-12-08 14:25:51 +00:00
f1ori
5f0f6b71b4 * revert last commit, something is more broken than before 
* UTC timestamps and lastseen-properteries still needs some debugging


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6519 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-12-06 21:54:32 +00:00
f1ori
8c8b642eba * fix timezone problem 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6518 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-12-06 21:03:12 +00:00
orbiter
4782d2c438 fix for search bug that appeared when looking at page 3 of results or further 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6515 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-12-03 12:25:03 +00:00
orbiter
29fde9ed49 better control of ranking order in sort stack 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6514 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-12-03 00:36:07 +00:00
orbiter
66923ebc6c - modified method in RequestHeader that delivers the host name of requester: no more reverse domain lookup (may have killed interface performance in some cases) 
- added logging output for shutdown servlet: show ip of requester of the shutdown

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6512 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-12-02 22:56:28 +00:00
orbiter
e34e63a039 preset of proper HashMap dimensions: should prevent re-hashing and increase performance 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6511 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-12-02 14:01:19 +00:00
orbiter
4a5100789f replaced _all_ size() == 0 with isEmpty() and all size() > 0 with !isEmpty(). The isEmpty() method is much faster in some cases, especially when used to access badly balanced hashtables where an size() operation becomes a large iteration. 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6510 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-12-02 00:37:59 +00:00
orbiter
f4946eaf27 - better thread dump 
- suppressed one server exception

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6509 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-12-01 22:53:36 +00:00
orbiter
9743b70d1c disabled keep-alive of server, not really needed for speed but a cause for much trouble and memory occupancy 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6508 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-12-01 19:14:16 +00:00
orbiter
491ba6a1ba - some refactoring in workflow 
- some refactoring in search process
- fixed image search for json and rss output
- search navigation on bottom of search result page in cases where there are more than 6 results on page
- fixes for number of displayed documents
- disabled pseudostemming

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6504 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-11-24 11:13:11 +00:00
orbiter
969123385b added json and rss output for image search 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6503 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-11-23 16:10:50 +00:00
orbiter
d183f8d980 refactoring (moved code from ContentTransformer to TemplateEngine) 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6498 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-11-20 14:57:00 +00:00
orbiter
23aef43786 - better synchronization in SortStack 
- better ThreadGroup organization
- less worker threads for media search (64 was too much...)


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6497 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-11-20 14:35:33 +00:00
orbiter
7b1f5b0430 - better media search ranking 
- better concurrency with enhanced synchronization in sort stack

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6496 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-11-20 13:19:12 +00:00
orbiter
4df88a4e7a - fixes for missing or bad hashCode computation 
- fixes for bad equals() methods that had not been used by hash maps and therefore some classes did not work as objects in hash maps.
- this may also affect some cases where double-checks should have been, but did not work.

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6495 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-11-20 12:11:56 +00:00
orbiter
dbdf2570ba added comparator and more fixes for SortStack/SortStore 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6494 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-11-20 03:30:48 +00:00
orbiter
1dff620181 Better implementation of SortStack and SortStore and adoptions in all using classes to implement the necessary Comparable interface and hash code computation. 
The better SortStack performance affects crawling and image search speed and quality.

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6492 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-11-19 13:49:28 +00:00
orbiter
fe41a84330 some enhancements in web caching: avoid double loading of response metadata and/or content 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6491 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-11-19 10:17:26 +00:00
orbiter
06d0dcde20 more enhancements to image search 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6490 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-11-19 00:43:42 +00:00
orbiter
4c6312d103 enhanced image search 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6489 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-11-18 23:56:05 +00:00
orbiter
2d8f3ee301 some performance hacks 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6488 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-11-18 16:03:28 +00:00
orbiter
94b2a664f3 - use a static DiskFileItemFactory (one instantiation is enough) 
- use more memory for the DiskFileItemFactory to avoid IO when POST commands come

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6485 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-11-18 15:05:51 +00:00
orbiter
013f337d3f - avoid unnecessary host name lookups for localhost 
- avoid unnecessary reverse domain name lookups for remote access

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6481 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-11-16 23:00:54 +00:00
orbiter
20c5d78a5c fix for a ConcurrentModificationException 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6478 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-11-11 23:31:12 +00:00
orbiter
7144d2df6e added crawlReceipt servlet as individual class to examine OOM problem as documented in 
http://forum.yacy-websuche.de/viewtopic.php?p=18120#p18120

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6476 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-11-11 16:12:00 +00:00