Marown/yacy_search_server
1
0
Fork 0
mirror of https://github.com/yacy/yacy_search_server.git synced 2024-09-22 00:00:59 +02:00
Code Issues Actions Packages Projects Releases Wiki Activity
5,428 Commits 1 Branch 20 Tags 836 MiB
0f0b4aec75
Commit Graph

5428 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
orbiter
0f0b4aec75 better index cell merge logic 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5754 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-03-30 06:22:27 +00:00
orbiter
832fef670f migration of urls-files into subdirectory METADATA 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5753 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-03-30 04:41:06 +00:00
orbiter
fa07234d4e fix for clear method: now deletes files 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5752 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-03-29 21:28:14 +00:00
apfelmaennchen
eb65990f85 small fix for opera in yacyui-portalsearch 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5751 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-03-29 12:49:42 +00:00
apfelmaennchen
695c420bcd small fix for yacyui-portalsearch 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5750 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-03-29 10:31:15 +00:00
apfelmaennchen
95885a263a - added default properies to yacyui-portalsearch 
- see http://localhost.8080/yacy/ui/yacyui-portaltest.html

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5749 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-03-29 09:29:08 +00:00
apfelmaennchen
c001a020a9 - small modifications to yacyui-portalsearch 
- see http://forum.yacy-websuche.de/viewtopic.php?f=15&t=1762&p=13459#p13459

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5748 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-03-28 23:06:15 +00:00
lulabad
df87e4dbf6 missing count of send Index and URLs 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5747 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-03-28 20:49:58 +00:00
apfelmaennchen
34a825f90d small fix for yacyui-portaltest.html 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5746 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-03-27 06:41:39 +00:00
apfelmaennchen
9f9d7f875d small fix 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5745 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-03-27 06:30:59 +00:00
apfelmaennchen
453f3aaa94 RichClient: further clean-up 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5744 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-03-27 06:06:14 +00:00
apfelmaennchen
e888c9a934 RichClient: 
- renamed base theme to start theme
- reoved all but start theme
- additional themes can be downloaded from http://jquery-ui.googlecode.com/files/jquery-ui-themes-1.7.zip
- or a custom theme can be generated at http://jqueryui.com/themeroller/
- themes are installed into DATA/LOCALE/htroot/yacy/ui/css/themes
- update for RichClient theme selection will follow soon

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5743 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-03-27 06:04:39 +00:00
apfelmaennchen
42c5f930c8 reverted an accidental commit 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5742 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-03-24 05:41:29 +00:00
apfelmaennchen
b5e6232f8d small correction of font-size for portal search 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5741 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-03-23 21:26:28 +00:00
apfelmaennchen
7425c6c3ca added an ajax loading graph to portal search 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5740 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-03-23 20:22:53 +00:00
apfelmaennchen
a975ae4a7e Added YaCy portal search: http://localhost:8080/yacy/ui/yacyui-portaltest.html 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5739 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-03-23 20:16:07 +00:00
apfelmaennchen
b57a1820bd small fix for jquery-faviconize-1.0.js to handle https properly 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5738 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-03-21 19:35:20 +00:00
apfelmaennchen
075b58a0a9 minor fixes for RichClient search 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5737 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-03-21 18:54:42 +00:00
borg-0300
c450e3746b svn attributes added 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5736 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-03-20 15:44:59 +00:00
orbiter
37f892b988 added new concurrent merger class for IndexCell RWI data 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5735 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-03-20 14:54:37 +00:00
borg-0300
8c494afcfe svn attributes added 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5734 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-03-20 11:21:32 +00:00
orbiter
67aaffc0a2 - added Latency control to the crawler: 
because of the strongly enhanced indexing speed when using the new IndexCell RWI data structures (> 2000PPM on my notebook), it is now necessary to control the crawling speed depending on the response time of the target server (which is also YaCy in case of some intranet indexing use cases).
The latency factor in crawl delay times is derived from the time that a target hosts takes to answer on http requests. For internet domains, the crawl delay is a minimum of twice the response time, in intranet cases the delay time is now a halve of the response time.

- added API to monitor the latency times of the crawler:
a new api at /api/latency_p.xml returns the current response times of domains, the time when the domain was accessed by the crawler the last time and many more attributes.

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5733 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-03-20 10:21:23 +00:00
lotus
7426dde6a6 windows installer: 
* install 64 bit JRE in case of 64 bit OS (not testet yet)
* less languages but localized hint boxes
* some cosmetics

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5732 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-03-19 18:53:30 +00:00
orbiter
0926310461 another performance hack 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5731 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-03-18 22:33:36 +00:00
orbiter
ebe5d69d14 performance hacks 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5730 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-03-18 22:19:08 +00:00
orbiter
61f9dbf0cc - fixed a display problem in watch crawler 
- another small enhancement in balancer

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5729 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-03-18 21:25:52 +00:00
orbiter
b3f75e48fa - enhanced balancer: auto-solving of waiting-deadlocks 
- removed deprecated cache-init size value
- more debug lines for IndexCell cache dump merge

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5728 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-03-18 20:21:19 +00:00
orbiter
9a90ea05e0 added a merge operation for IndexCell data structures 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5727 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-03-18 16:14:31 +00:00
orbiter
d99ff745aa fix for http://forum.yacy-websuche.de/viewtopic.php?p=13378#p13378 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5726 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-03-18 10:29:13 +00:00
orbiter
0c3ab291c4 fix for http://forum.yacy-websuche.de/viewtopic.php?p=13354#p13354 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5725 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-03-17 22:20:58 +00:00
orbiter
a9cea419ef Integration of the new index data structure IndexCell 
This is the start of a testing phase for IndexCell data structure which will replace
the collections and caching strategy. IndexCall creation and maintenance is fast, has
no caching overhead, very low IO load and is the basis for the next data structure,
index segments.

IndexCell files are stored at DATA/<network>/TEXT/RICELL
With this commit still the old data structures are used, until a flag in yacy.conf is set.
To switch to the new data structure, set
useCell = true
in yacy.conf. Then you will have no access any more to TEXT/RICACHE and TEXT/RICOLLECTION

This code is still bleeding-edge development. Please do not use the new data structure for
production now. Future versions may have changed data types, or other storage locations.
The next main release will have a migration feature for old data structures.

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5724 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-03-17 13:03:27 +00:00
borg-0300
fd0976c0a7 refactoring 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5723 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-03-16 18:08:43 +00:00
orbiter
83792d9233 more refactoring 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5722 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-03-16 16:24:53 +00:00
borg-0300
ce79239322 "typo" 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5721 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-03-16 16:22:33 +00:00
borg-0300
cdbdc731c5 small updates: unescape, isCGI 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5720 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-03-16 08:49:49 +00:00
orbiter
474aac65af more refactoring 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5719 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-03-16 08:32:28 +00:00
orbiter
209f25f5f5 refactoring to integrate indexCell data structures 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5718 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-03-16 00:18:37 +00:00
borg-0300
359a238acf faster isCGI() 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5717 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-03-14 13:47:49 +00:00
borg-0300
f75628e53b some corrections 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5716 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-03-14 11:08:32 +00:00
orbiter
b7138e5fcb even more efficient comparator calls (less System.arraycopy for primary keys) 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5715 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-03-14 00:41:20 +00:00
orbiter
65784eb656 - more efficient comparator calls 
- fix for http://forum.yacy-websuche.de/viewtopic.php?p=13331#p13331

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5714 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-03-14 00:07:37 +00:00
orbiter
44874cb550 added a deleteOnExit for blob file deletion in case that a deletion is not successful. 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5713 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-03-13 22:47:31 +00:00
orbiter
66f78d67e0 bad idea. Concurrency in index management will be done differently 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5712 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-03-13 22:22:11 +00:00
orbiter
7dff1cba62 removed option to use different primary keys in kelondro tables 
this option was never used and there is also no use to set other columns but the first as the primary key. as a result, access methods to the key do not need to compute key positions, and they work faster.

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5711 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-03-13 16:52:31 +00:00
orbiter
7f67238f8b refactoring of plasmaWordIndex: less methods in the class, separated the index to CachedIndexCollection 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5710 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-03-13 14:56:25 +00:00
orbiter
14a1c33823 refactoring of wordIndex class 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5709 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-03-13 10:34:51 +00:00
orbiter
d49238a637 more performance hacks: better default values for scaling, less memory usage 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5708 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-03-13 10:07:04 +00:00
orbiter
39644dc14e performance hacks to compare methods in database core 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5707 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-03-13 09:30:19 +00:00
orbiter
e2e7949feb replaced old PPM computation with a better one that simply sums up events that had been stored in the profiling table. 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5706 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-03-13 00:13:47 +00:00
orbiter
f6d989aa04 added new class RowSetArray which arranges RowSet objects like Elements in a hashtable, but still provides the functionality of sorted enumeration. The new class is now integrated into the ObjectIndexCache, which is the core class to provide index functions to all database files. The new index access is about twice as fast as before. This has strong speed enhancement effects on all parts of YaCy. 
The speed of the kelondro indexing class ObjectIndexCache can be compared with Javas standard TreeMap with the main method in IntegerHandleIndex. The result is, that the kelondro indexing needs only 1/5 of the memory that TreeMap uses! In exchange, the kelondro classes are slower than TreeMap, about four (!) times slower. However, this is not so bad because the better use of the memory is a strong advantage and makes it possible that YaCy can maintain such a large number of document (> 50 million) in one peer.

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5705 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-03-12 23:05:18 +00:00