Marown/yacy_search_server
1
0
Fork 0
mirror of https://github.com/yacy/yacy_search_server.git synced 2024-09-21 00:00:13 +02:00
Code Issues Actions Packages Projects Releases Wiki Activity
6,414 Commits 1 Branch 20 Tags 836 MiB
3300930fc5
Commit Graph

141 Commits

Author SHA1 Message Date
orbiter
3300930fc5 - (almost) fixed FTP crawler 
- integrated/fixed SMB crawler

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6742 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2010-03-11 15:43:06 +00:00
orbiter
57e1eae95e longer time-out for url fetching .. may help to show all that links that the statistic say for a search result 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6727 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2010-03-07 22:23:08 +00:00
orbiter
f561e340c6 show more results of single domains when not authorized fully (up to 100) 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6720 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2010-03-07 00:12:58 +00:00
orbiter
884b262130 - added a new Wiki Namespace Navigator 
- some redesign of Navigator data structures

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6716 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2010-03-05 21:25:49 +00:00
orbiter
617dfbbd06 allo 'authorization by encoded password' also if requesting client is not from localhost but from the same host as yacy is running on. 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6714 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2010-03-05 16:03:55 +00:00
orbiter
727dd9b193 - fixed a bug in robots.txt parser 
- moved storage of robots.txt entries to WorkTables, so it is now possible to browse the robots entries with the table browser

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6710 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2010-03-04 11:58:07 +00:00
orbiter
54af9e6b49 - added parsing of robots meta-tag in html headers to detect a noindexing request 
- added evaluation and indexing prevention in case that a noindexing is given in a html file

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6709 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2010-03-03 23:32:56 +00:00
orbiter
f175f9a2d3 changed way how number of search requests are counted: 
so far only search requests at the remote search interface had been counted.
This was done to protect the privacy of searchers, because counting was not done and published at the own search interface.
This caused that no search requests of robinson peers had been counted, becuase they cannot be counted at remote peer.
This change introduces a distinction of locally done search requests at the local search interface from search requests that are on the local interface but had been submitted from a remote IP without authentication.
Now 3 counters are maintained:
- partial count of remote searches
- total count of local searches on robinson peers from non-authenticated clients
- total count of local searches on robinson peers from localhost or authenticated clients
In the global statistic of search requests now the first two counters of the three cases are added
Because we habe a large number of robinson peers with a large number of remote non-authenticated requests the statistic should show at least three times of the number of search requests.


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6696 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2010-02-24 13:53:55 +00:00
orbiter
8030ed3319 self-healing for lost crawl profile handles 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6680 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2010-02-18 21:55:45 +00:00
orbiter
e3e5e05ec2 fix for problem in ranking setting which was caused by the introduction of a toString() method in serverObjects 
see also: http://forum.yacy-websuche.de/viewtopic.php?p=19310#p19310

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6678 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2010-02-17 21:31:08 +00:00
orbiter
ef62d017e5 integrated session id filtering for crawler 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6672 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2010-02-15 23:15:17 +00:00
orbiter
d8d9984913 added framework for session id filtering (not ready yet) 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6671 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2010-02-15 22:30:41 +00:00
lotus
945e0ba5a5 allow global search if res. observer disabled index transmission 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6658 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2010-02-09 17:14:16 +00:00
orbiter
74e736c903 missing file for last commit 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6645 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2010-02-04 14:52:58 +00:00
orbiter
d77782a8d5 removed bookmark tags file, tags are now stored only in RAM 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6638 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2010-02-01 22:44:59 +00:00
orbiter
a131ebbcb5 one more fix for NPE, see 
http://forum.yacy-websuche.de/viewtopic.php?p=19010#p19010

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6634 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2010-02-01 11:28:37 +00:00
orbiter
24060885b6 - added Tables abstraction in data.Tables.java 
fix for
http://forum.yacy-websuche.de/viewtopic.php?p=18910#p18910
http://forum.yacy-websuche.de/viewtopic.php?p=18894#p18894
http://forum.yacy-websuche.de/viewtopic.php?p=18814#p18814


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6631 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2010-01-29 18:02:09 +00:00
orbiter
7fdf59a77f misc NPE check 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6630 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2010-01-29 15:59:24 +00:00
orbiter
a512aef6ad fix for http://forum.yacy-websuche.de/viewtopic.php?p=18918#p18918 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6629 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2010-01-29 10:33:20 +00:00
orbiter
69c29acb6e no exception thread dump if parser cannot parse becuase that mime-type/extension is in the deny-set 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6611 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2010-01-22 13:21:37 +00:00
orbiter
8ce936bcdd added an api recording function: it shall be possible to record 
all operations on YaCy in a database that should make it possible
1) to re-create a setting on fresh peers
2) to transmit a setting from one peer to another
3) to re-create crawl starts after a complete deletion of the index
This functionality will also support
4) scheduled re-crawls (new implementation)
To implement this, a new database structure has been crated that stores maps into blob heaps. to encode maps the b-encoding technique was used (this is the same encoding that torrent files use)
- added a b-encoder
- enhanced the b-decoder
- added a b-encoded map heap data structure
- added a table organisation based on b-encoded heaps
- added a servlet to maintain such tables (see Tables_p.html)
- integrated the servlet into the Advanced Settings menu
- added an api recording based on the new tables

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6606 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2010-01-21 22:06:03 +00:00
orbiter
234f733a3d - relocation of seed db is better for network switch than re-initialization because of the embedding of the peers object in other objects 
- small refactoring of blacklist interface code to remove PMD warnings


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6593 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2010-01-18 00:07:20 +00:00
orbiter
473b11033d fixed network switch process - crawling did not work after a switch before this fix 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6592 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2010-01-17 23:33:15 +00:00
orbiter
fd7b348973 some fixes for the network switch 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6591 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2010-01-17 22:07:08 +00:00
orbiter
f6731c6240 more logging etc. 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6589 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2010-01-17 00:41:50 +00:00
orbiter
938e806182 tried to fix date problem that may have prevented that foreign peers stay in the network 
- removed unused code
- removed possibly wrong utc difference correction

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6581 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2010-01-13 20:01:46 +00:00
orbiter
82f57f79e5 more PMD enhancements 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6576 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2010-01-13 00:23:07 +00:00
orbiter
5d930c96f0 more fixes to search result page navigation 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6575 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2010-01-13 00:04:37 +00:00
orbiter
8c520f128d reverted a change in ranking process committed this afternoon 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6573 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2010-01-12 20:56:37 +00:00
orbiter
a06f7ddb33 more PMD recommendations 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6572 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2010-01-12 20:53:19 +00:00
orbiter
18172451a0 better search computation: 
- increased sort limit, now 3000 entries, before: 1000
  this should cause that more results can be shown in case
  of strong limitating constraints, like domain navigation
- enhanced the sort process
- check against domain navigator bugs
- fix in sort stack
- showing now all naviagtion pages at first search (not only next page)


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6569 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2010-01-12 15:01:44 +00:00
orbiter
66c0a8e849 more PMD recommendations 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6567 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2010-01-11 22:18:38 +00:00
orbiter
dd459281c8 applied code changes that are recommended by PMD 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6563 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2010-01-10 23:09:48 +00:00
orbiter
d77a8f3b3e added some modifications recommended by PMD for better performance 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6560 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2010-01-10 01:40:26 +00:00
orbiter
bb2e03761c - fix for deadlock with 100% CPU during search 
- fix for failure of ranking because of a ConcurrentModificationException

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6553 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2010-01-07 12:41:43 +00:00
orbiter
dff4f95c78 some patches to get the torrent parser working 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6551 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2010-01-07 00:42:12 +00:00
low012
82198acc06 *) minor changes 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6537 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-12-28 11:06:49 +00:00
orbiter
57d729e377 fix for negative numbers in network statistic 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6532 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-12-21 11:36:48 +00:00
orbiter
d548bd41ad fix for a npe during search 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6528 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-12-15 12:15:39 +00:00
orbiter
a37878b7d5 url parser regex performance hack 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6524 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-12-10 14:40:32 +00:00
orbiter
b527d2ebfa fix for media search 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6522 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-12-09 23:47:37 +00:00
orbiter
362b7a929b added extensive memory protection logic to avoid out of memory errors that may be caused by the RowCollection memory allocation function 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6521 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-12-09 23:27:26 +00:00
orbiter
8281e29963 - more configuration for profiling graph (number of events) 
- more logging for a shutdown: print reason and accessing IP into log


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6520 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-12-08 14:25:51 +00:00
orbiter
4782d2c438 fix for search bug that appeared when looking at page 3 of results or further 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6515 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-12-03 12:25:03 +00:00
orbiter
29fde9ed49 better control of ranking order in sort stack 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6514 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-12-03 00:36:07 +00:00
orbiter
e34e63a039 preset of proper HashMap dimensions: should prevent re-hashing and increase performance 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6511 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-12-02 14:01:19 +00:00
orbiter
4a5100789f replaced _all_ size() == 0 with isEmpty() and all size() > 0 with !isEmpty(). The isEmpty() method is much faster in some cases, especially when used to access badly balanced hashtables where an size() operation becomes a large iteration. 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6510 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-12-02 00:37:59 +00:00
orbiter
491ba6a1ba - some refactoring in workflow 
- some refactoring in search process
- fixed image search for json and rss output
- search navigation on bottom of search result page in cases where there are more than 6 results on page
- fixes for number of displayed documents
- disabled pseudostemming

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6504 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-11-24 11:13:11 +00:00
orbiter
969123385b added json and rss output for image search 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6503 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-11-23 16:10:50 +00:00
orbiter
23aef43786 - better synchronization in SortStack 
- better ThreadGroup organization
- less worker threads for media search (64 was too much...)


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6497 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-11-20 14:35:33 +00:00