Marown/yacy_search_server
1
0
Fork 0
mirror of https://github.com/yacy/yacy_search_server.git synced 2024-09-21 00:00:13 +02:00
Code Issues Actions Packages Projects Releases Wiki Activity
8,543 Commits 1 Branch 20 Tags 836 MiB
c63c3a4495
Commit Graph

1181 Commits

Author SHA1 Message Date
cominch
65c5826d93 bugfix 
Conflicts:
	source/net/yacy/document/parser/augment/AugmentParser.java
 2012-06-10 13:11:54 +02:00
cominch
aa0295917c augmentation 
Conflicts:
	source/net/yacy/interaction/AugmentHtmlStream.java
 2012-06-10 13:10:21 +02:00
cominch
ed2ea0f08e augmented browsing modification 
Conflicts:
	htroot/interaction/OverlayInteraction.html
	source/net/yacy/interaction/AugmentHtmlStream.java
 2012-06-10 13:07:57 +02:00
cominch
6b32f7c1f6 re-enable augmented proxy 2012-06-10 13:04:13 +02:00
cominch
3b08edec2e bugfix 
Conflicts:
	source/net/yacy/interaction/AugmentHtmlStream.java
 2012-06-10 13:03:56 +02:00
cominch
5f8ba7f4f2 small changes 
Conflicts:
	source/net/yacy/document/parser/augment/AugmentParser.java
	source/net/yacy/interaction/Interaction.java
 2012-06-10 13:02:00 +02:00
cominch
300b235ce8 Updated Demo Servlet 
Conflicts:
	htroot/About.html
	htroot/DemoServlet.html
	htroot/DemoServlet.java
	htroot/interaction/interaction.js
	source/net/yacy/interaction/Interaction.java
 2012-06-10 12:58:29 +02:00
cominch
90512640bf Added config switches for custom parser 
Conflicts:
	source/net/yacy/document/TextParser.java
 2012-06-10 12:49:36 +02:00
cominch
df47f31235 interaction: add special table interaction 
Conflicts:
	source/net/yacy/interaction/Interaction.java
 2012-06-10 12:43:01 +02:00
cominch
e14f2881ae interaction: add special table interaction 
Conflicts:
	source/net/yacy/interaction/Interaction.java
 2012-06-10 12:41:16 +02:00
cominch
d7326079a8 interaction: add global variable store 
Conflicts:
	source/net/yacy/interaction/Interaction.java
 2012-06-10 12:39:27 +02:00
cominch
4e4e7a99f8 interaction: add global variable store 
Conflicts:
	source/net/yacy/interaction/Interaction.java
 2012-06-10 12:34:36 +02:00
cominch
bde07ed7a8 Add tagging overlay element 
Conflicts:
	htroot/env/templates/jqueryheader.template
	htroot/yacysearchitem.java
	source/net/yacy/interaction/Interaction.java
 2012-06-10 12:28:50 +02:00
cominch
b0bc0b4572 Add new demonstration module for client-side key-value store (backend: 
triplestore): /DemoServletInteraction.html

Conflicts:
	source/net/yacy/interaction/Interaction.java
 2012-06-10 10:53:30 +02:00
cominch
c9dc6cda02 Demonstration: include value from interaction in search results 
Conflicts:
	htroot/interaction/OverlayInteraction.html
	htroot/yacysearchitem.java
 2012-06-10 10:51:53 +02:00
cominch
bcbd8eee33 Add several parsers, for RDFa and rdf files. 
Conflicts:
	source/net/yacy/document/TextParser.java
 2012-06-10 10:42:33 +02:00
cominch
9ef5a80f4e add interaction for triples and selector for augmented browsing 
Conflicts:
	htroot/interaction/interaction.js
	source/net/yacy/interaction/Interaction.java
 2012-06-10 10:38:54 +02:00
cominch
282c1620d6 Allow TripleStore to be persistent after reboot 2012-06-10 10:36:16 +02:00
cominch
5d20cd324a Add Triplestore and RDF query interface 
Conflicts:
	build.xml
	defaults/yacy.init
	source/net/yacy/interaction/AugmentHtmlStream.java
 2012-06-10 10:35:59 +02:00
cominch
bc9a618e0a augmented browsing: ignore js and css, integrate more user interaction 
Conflicts:
	htroot/interaction/Footer.html
	source/net/yacy/interaction/AugmentHtmlStream.java
 2012-06-10 10:29:15 +02:00
cominch
b21048892b augmentedParser add features and integrate external html parser to 
modify existing web pages

Conflicts:
	addon/YaCy.app/Contents/Info.plist
	build.xml
 2012-06-10 10:23:35 +02:00
cominch
9cbfc1a1c0 augmentedProxy, which forwards every proxy request to a 
rewrite engine to customize existing webpages. originally implemented by
Florian Richter.

Conflicts:
	source/de/anomic/http/server/HTTPDProxyHandler.java
 2012-06-10 10:15:34 +02:00
Michael Peter Christen
3b992e6b00 using utf8 String compression in Webstructure database 2012-06-09 11:00:33 +02:00
Michael Peter Christen
26301a538d bugfix in Domains - dns-lookup 2012-06-09 10:59:45 +02:00
Michael Peter Christen
cde20911bb saved a bit more ram using UTF8 String compression for OpenGeoDB and 
Geonames data files.
 2012-06-09 10:07:11 +02:00
Michael Peter Christen
225ee42879 made the GeoLocation into an interface with the current 
integer implementation as accuracy implementation of 1.863cm
 2012-06-09 09:46:27 +02:00
Michael Peter Christen
2280a7b276 - changed initialization order to prefer allocation of memory for table 
files first
- bugfixes in memory amount calculation
 2012-06-09 09:05:47 +02:00
Michael Peter Christen
0746308bc2 only the metadata tables shall be able to use the tail cache 2012-06-08 18:36:11 +02:00
Michael Peter Christen
7ec9bef0c3 fix for OOM 2012-06-08 17:14:09 +02:00
Michael Peter Christen
41c02cb10e - less restrictions for usage of Table RAM copy 
- new limit to use the table copy (instead of flag): 400MB available. If
less is available, then a copy is never used. If more is available, then
it can be used if there is a remaining space of at least 200MB
- flush caches more often: flush the Digest cache
 2012-06-08 12:48:25 +02:00
Michael Peter Christen
b8f56a9803 npe bugfix 2012-06-08 10:20:43 +02:00
Michael Peter Christen
dd14b19c26 lazy initialization of block rank table ... only normal web search uses 
this. When interactive search or location search is used, the block rank
is switched off
 2012-06-08 09:41:29 +02:00
Michael Peter Christen
ba10caf89a lazy initialization of database tables 2012-06-08 09:30:51 +02:00
Michael Peter Christen
701b9a28a0 Merge branch 'master' of ssh://git@gitorious.org/yacy/rc1.git 
Conflicts:
	htroot/PerformanceMemory_p.java
 2012-06-08 09:16:16 +02:00
Michael Peter Christen
ab7107b34b fixed RWIProcess queue limits: now discovering hidden results for mass 
result retrieval
 2012-06-08 09:14:54 +02:00
Michael Peter Christen
10c9c17d51 fixed handlemap spread factor and null iterator handling 2012-06-08 09:13:41 +02:00
Michael Peter Christen
b0095c8d3c flush the compressor cache when a cleanup is done 2012-06-07 19:42:33 +02:00
Michael Peter Christen
a61f44f9e4 lazy initialization of block rank table. 
this causes that the table is not initialized when there is no search is
done. the effect is most strong if YaCy is started headless which causes
no browser pop-up which otherwise would load the search page and
therefore trigger the initialization of the table.
 2012-06-07 13:16:38 +02:00
Michael Peter Christen
96e9d77270 Merge branch 'master' of ssh://git@gitorious.org/yacy/rc1.git 
Conflicts:
	source/net/yacy/cora/sorting/WeakPriorityBlockingQueue.java
 2012-06-06 20:13:28 +02:00
Michael Peter Christen
00f2df1120 a variety of possible memory leak fixes 2012-06-06 18:23:18 +02:00
Michael Peter Christen
3dd8376825 added automatic cleaning of cache if metadata and file database size is 
not equal. It might happen that these data is different because one of
that caches is cleaned after a while or when it is too big. The metadata
is then not cleaned, but now wiped after a checkup process at every
application start. This should cause a bit less memory usage.
 2012-06-06 14:15:24 +02:00
Michael Peter Christen
d0ec8018f5 fixes for bad long computation 2012-06-06 14:13:31 +02:00
Michael Peter Christen
6bb07afcc3 accept also files with other file prefix; used to read 'foreign' cache 
files
 2012-06-06 13:36:10 +02:00
Michael Peter Christen
96c8119b50 added GeoLocation / GeoPoint classes which uses less memory than 
Location/Coordinates and has initializers with correct order of lat,lon
coordinates
 2012-06-06 12:57:42 +02:00
Michael Peter Christen
461a0ce052 removed warnings 2012-06-05 20:03:43 +02:00
Michael Peter Christen
62ae9bbfda allow more POIs, get more at once 2012-06-05 18:29:54 +02:00
Michael Peter Christen
407fdf6968 more bug fixes and performance hacks for search process 2012-06-05 15:04:23 +02:00
Michael Peter Christen
a1fe65b115 performance hacks 2012-06-05 12:06:26 +02:00
Michael Peter Christen
2fe207f813 Merge branch 'master' of ssh://git@gitorious.org/yacy/rc1.git 2012-06-04 23:44:38 +02:00
Michael Peter Christen
5aee19daa4 added show from cache in search results (not yet finished) 2012-06-04 23:44:26 +02:00
Michael Peter Christen
5e562dcdb7 adopted vocabulary usage within anotation/naviagtion feature of search 
to new SimpleVocabulary class
 2012-06-04 23:43:30 +02:00
Michael Peter Christen
514700291a moved Vocabulary to cora package (added in git 
964406ad17)
 2012-06-04 23:41:36 +02:00
Michael Peter Christen
0284a4d88f more fixes for double precision of coordinates 2012-06-04 23:37:41 +02:00
Michael Peter Christen
964406ad17 added concurrency enhancement to xml parser 2012-06-04 23:35:56 +02:00
Michael Peter Christen
240045cf7c fix for bad distance computation 2012-06-04 16:33:16 +02:00
Michael Peter Christen
e0d8643226 - performance hacks 
- added log warnings in case that search processes run into time-out
situations
- better concurrency for Integer formatter (used a non-synchronized
formatter before)
- bugfix for search termination (a poison pill was missing)
- added timeout parameters for search (again) -> target is, that they
are never reached.
 2012-06-04 15:37:39 +02:00
Michael Peter Christen
7a329465b3 using pre-compile pattern in blacklist; should enhance search speed 2012-06-04 15:34:53 +02:00
Michael Peter Christen
6e83b02b83 - bugfix for surrogate file reader 
- bugfix for location search: suppress empty search
 2012-06-01 00:08:31 +02:00
Michael Peter Christen
9b4c699526 ehanced location search: 
- search request are now made using a map boundary
- search results are only computed for the map boundary
- the number of results is adopted to the results in the visible range
- added a double-buffering for the search result markers
- added a search query option for the search results:
/radius/<lat>/<lon>/<radius>
 2012-05-31 22:39:53 +02:00
Michael Peter Christen
834dc6b263 store more data from interface access 2012-05-31 00:47:07 +02:00
Michael Peter Christen
1f48d1528b performance hacks 2012-05-31 00:46:30 +02:00
Michael Peter Christen
c70aaccdc9 better location to generate a guid for rss messages 2012-05-30 17:14:25 +02:00
Michael Peter Christen
10da7335ea performance hack: use a hash cache for all hashes that are computed by a 
byte array. If this hash is used in a HashMap (which is very often the
case) then this hack eliminates a lot of re-computations of the same
hash.
 2012-05-30 16:59:13 +02:00
Michael Peter Christen
f8a0cf6d7c RSSMessages do not need a concurrent hash map -> removed overhead 2012-05-30 16:44:03 +02:00
Michael Peter Christen
07ca7e4dd1 enhanced RSS parsing by ensuring that it is parsed with a buffered input 
stream
 2012-05-30 16:40:37 +02:00
Michael Peter Christen
7c1feefb28 introduced a default 10 second time-out in rwi normalization time 
uring search process to prevent endless deadlocks after a very long
running search
 2012-05-30 16:26:05 +02:00
Michael Peter Christen
8d997d55b6 better logging 2012-05-30 15:47:35 +02:00
Michael Peter Christen
65d37e6a20 only ASCII needed in seed bitflags 2012-05-30 15:42:28 +02:00
Michael Peter Christen
0f82fb3628 using double instead float for a better release ordering 2012-05-30 15:28:20 +02:00
Michael Peter Christen
43c2c6e588 better logging 2012-05-30 15:27:45 +02:00
sixcooler
56087c1f23 bump to httpclient- httpcore-, httpmime- 4.2 2012-05-30 14:46:21 +02:00
Michael Peter Christen
71c3163f3d - fixes to node identification 
- added link to node in network list
- added marking of portal search node peers
 2012-05-29 01:38:54 +02:00
Michael Peter Christen
4d3cc02168 replaced old bzip2 library against better documented commons-compress 
package from http://commons.apache.org/compress/
 2012-05-28 23:53:48 +02:00
Michael Peter Christen
ad222be7f8 added node state icon in network list 2012-05-25 17:29:54 +02:00
Michael Peter Christen
3c2bec681f added a root node flag: identifies peers with short ping time 2012-05-25 15:33:02 +02:00
Michael Peter Christen
c846e9ca14 redesign of the crawler monitor page: show crawled pages instead of 
queue of urls that shall be crawled
 2012-05-25 01:45:38 +02:00
Michael Peter Christen
c15fcde1c8 add-on to latest commit 2012-05-21 17:52:30 +02:00
Michael Peter Christen
cf47d94888 performance hack to parse numbers inside of substrings without actually 
generating a substring. This avoids the allocation of a String object
ech time a substring is parsed. Should affect CPU load during RWI
transmission.
 2012-05-21 13:40:46 +02:00
Michael Peter Christen
7e0ddbd275 added a "fromCache" flag in Response object to omit one cache.has() 
check during snippet generation. This should cause less blockings
 2012-05-21 03:03:47 +02:00
Michael Peter Christen
81737dcb18 removed stack trace from swf parser since we cant do anything there 2012-05-21 02:27:06 +02:00
Michael Peter Christen
7bf421b9dd - fixed image search page navigation 
- removed some deadlocks and ConcurrentModificationExceptions during
DidYouMean collection
 2012-05-21 01:58:29 +02:00
Michael Peter Christen
c6a09eab0b synchronization needed 2012-05-21 00:58:29 +02:00
Michael Peter Christen
fb94b47b1a changed queue sizes to have less memory occupied during indexing 2012-05-21 00:19:03 +02:00
Michael Peter Christen
76157dc2c3 bugfix for http://bugs.yacy.net/view.php?id=173 2012-05-21 00:18:00 +02:00
reger
6696cb1313 bugfix: lookup of peernames no result for active peer in page IndexControlRWIs_p.html -> Transfer RWI to other Peer 
SeedDB.lookupByName searche for lowercase peerNames, while MapColumnIndex.getIndex uses peername as is in the keyset.
Changed the index init to insert lowercase peer names as key
 2012-05-20 05:25:16 +02:00
Michael Peter Christen
c6558cba08 more classification bugs 2012-05-20 02:59:47 +02:00
Michael Peter Christen
082831b9d6 search contentdom was checked in wrong way - fixed 2012-05-20 01:23:02 +02:00
reger
ee553d971e correct typo in scripts_txt comment 2012-05-19 02:09:16 +02:00
Michael Peter Christen
f294f2e295 bugfix to http://bugs.yacy.net/view.php?id=181 
tried to make a bit less 'noise' to dns server

also included: less processes in snippet fetch to reduce load during
search on small computers
 2012-05-19 01:06:33 +02:00
Michael Peter Christen
acf8d521a2 fix for http://bugs.yacy.net/view.php?id=126 2012-05-19 00:21:03 +02:00
Michael Peter Christen
bb88878b4d the last commit was incomplete.. 2012-05-18 22:33:16 +02:00
Michael Peter Christen
d320a31ae1 bugfix for http://bugs.yacy.net/view.php?id=186 2012-05-18 22:18:47 +02:00
Michael Peter Christen
fa735f4f04 Merge branch 'master' of ssh://git@gitorious.org/yacy/rc1.git 2012-05-17 23:40:08 +02:00
Michael Peter Christen
3e1bc9477f Merge branch 'master' of ssh://git@gitorious.org/yacy/rc1.git 2012-05-17 13:58:09 +02:00
Michael Peter Christen
6f8a2fef1f small speed enhancement using a column factory 2012-05-17 11:08:48 +02:00
Roland 'Quix0r' Haeder
d10627d591 More sync in close() methods 
Conflicts:
	source/net/yacy/kelondro/logging/GuiHandler.java
	source/net/yacy/kelondro/workflow/InstantBusyThread.java
 2012-05-17 06:03:18 +02:00
Roland 'Quix0r' Haeder
b3ae2aa41f With or without 'final'? At least please try it in other methods 
Conflicts:
	source/de/anomic/tools/tarTools.java
 2012-05-17 06:00:49 +02:00
Roland 'Quix0r' Haeder
fbb946f913 Made a method static (Eclipse suggested it), removed unused import, pk=null check does now output a warning in logfile 2012-05-17 05:55:44 +02:00
Michael Peter Christen
52d307c735 prevent that the snippet fectch process removes catchall entries 2012-05-17 05:18:52 +02:00
Michael Peter Christen
7eece0256f moved yacy.logging to defaults according to request in 
http://bugs.yacy.net/view.php?id=55
 2012-05-17 04:26:03 +02:00