sixcooler
72561926aa
do not overwrite yacy.conf in case of an exception
...
may be a fix for http://mantis.tokeek.de/view.php?id=180
2014-10-15 18:13:54 +02:00
Michael Peter Christen
ec6082c872
very bad language detection hack fix hack
2014-09-05 23:29:09 +02:00
Michael Peter Christen
0ceeceb35e
more logic on Solr queries; usage of the query terms in posprocessing,
...
saving one query for double document detection now per document
2014-08-04 02:35:38 +02:00
orbiter
97983ba89f
fixed generics warnings for generic array instantiation that appeared
...
after migration to Java 7
2014-05-20 21:50:16 +02:00
orbiter
88f4af90da
removed warnings
2014-05-13 22:27:31 +02:00
Michael Peter Christen
17e0956312
refactoring of SystemLoad calls (only one backend tool)
2014-04-11 09:25:18 +02:00
Michael Peter Christen
8b44fcf0f4
added missing @Override annotation
2014-03-28 13:48:37 +01:00
Michael Peter Christen
acc8d7faa7
fixed setting of shortMemoryStatus in MemoryControl
2014-02-09 12:25:55 +01:00
sixcooler
40a4030b55
configurable max-load values for YaCy-Threads:
...
try lower values on smal systems like a Pi
2014-01-21 17:04:22 +01:00
Michael Peter Christen
2c39b65409
fixes for searches containing stopwords. The fix was done using a
...
reconstruction of the search word set access method to protect that
words are deleted from the sets from the outside of the QueryGoal class.
2013-11-26 02:24:47 +01:00
Michael Peter Christen
1a4a69c226
set more logger to 'final static'
2013-11-13 06:18:48 +01:00
Michael Peter Christen
5e31bad711
- the webgraph shall store all links which appear on a web page and not
...
all unique links! This made it necessary, that a large portion of the
parser and link processing classes must be adopted to carry a different
type of link collection which carry a property attribute which are
attached to web anchors.
- introduction of a new URL class, AnchorURL
- the other url classes, DigestURI and MultiProtocolURI had been renamed
and refactored to fit into a new document package schema, document.id
- cleanup of net.yacy.cora.document package and refactoring
2013-09-15 00:30:23 +02:00
Michael Peter Christen
cb85b22725
redesign of the image search process (with much better results,
...
unfortunately the index schema has changed and p2p image search will not
be muchmuch better until many people update)
2013-09-02 18:55:38 +02:00
Michael Peter Christen
47b1c81d08
- refactoring
...
- generalized writing of url attributes to solr documents
- added more url attributes to error documents
2013-08-20 15:46:04 +02:00
Roland Haeder
841a28ae76
Added 'final' for all exception blocks as this helps the Java compiler
...
to optimize memory usage
Conflicts:
source/net/yacy/search/Switchboard.java
2013-07-17 18:31:30 +02:00
Michael Peter Christen
5878c1d599
- refactoring of log to ConcurrentLog:
...
jdk-based logger tend to block
at java.util.logging.Logger.log(Logger.java:476) in concurrent
environments. This makes logging a main performance issue. To overcome
this problem, this is a add-on to jdk logging to put log entries on a
concurrent message queue and log the messages one by one using a
separate process.
- FTPClient uses the concurrent logging instead of the log4j logger
2013-07-09 14:28:25 +02:00
reger
7480e87386
- fix stopword handling for RWI see example http://bugs.yacy.net/view.php?id=247
...
- append language setting specific stopword list
- remove unused OVERHANG stack type
2013-06-06 22:07:54 +02:00
Michael Peter Christen
c1a2175fbc
added transparency to gif image animation and the integration to the
...
YaCy httpd for on-the-fly generated gifs (including animated gifs)
2013-04-21 12:29:05 +02:00
Marc Nause
75f9568472
*) only install files from the RELEASE directory
...
*) minor changes
2013-02-05 21:02:32 +01:00
Marc Nause
3bc5ee6e3d
*) added protection against CSRF in update download page
...
(http://localhost:8090/ConfigUpdate_p.html?releaseinstall=../../test.txt&deleteRelease=Delete+Release
does not work anymore)
2013-02-04 19:57:28 +01:00
Michael Peter Christen
5c0c56cfe1
Preparations to produce a click depth attribute in the search index.
...
This attribute can be used for ranking and for other purpose (demand by
customer)
The click depth is computed in two steps:
- during indexing the current fill-state of the reverse link index is
used to backtrack the current page to the root page. The length of that
backtrack is the clickdepth. But this does not discover the shortest
click depth. To get this, a second process to check again is needed
- added a process tag that can be used to do operations on the existing
index after a crawl; i.e. calculation the shortest clickpath. Added a
field to control this operation but not a method to operate on this.
- added a visualization of the clickpath length in the host browser
2013-01-02 20:55:43 +01:00
Michael Peter Christen
c5f67a5d6d
fixed a problem with local search from solr results: now all results
...
from solr are shown (again)
2012-11-01 10:22:22 +01:00
Michael Peter Christen
f8f05ecba7
- added a delete button in host browser to delete a complete subpath
...
- removed storage of default collection name - default is now "user"
- made stacking of crawl start points concurrently
2012-10-31 17:44:45 +01:00
Michael Peter Christen
f2d0418218
because the new PngEncoder had a problem with the PixelGrabber which is
...
caused by a JRE bug, the PixelGrabber had to be circumvented using an
own frame buffer which can be read without a PixelGrabber. This resulted
in ultra-fast and much less memory-consuming transformation. YaCy images
are now generated really fast!
2012-10-25 17:59:20 +02:00
Michael Peter Christen
b400fc7b4d
fix for file parser problem
2012-10-17 18:06:44 +02:00
Michael Peter Christen
6017691522
added an exception catch
2012-10-17 13:56:11 +02:00
Michael Peter Christen
613cf7da7f
enhancement to post argument parsing - possible fix to zero-filled
...
parameter values
2012-10-11 10:46:06 +02:00
Michael Peter Christen
a06930662c
replaced some more .getBytes() with UTF8/ASCII.getBytes()
2012-10-09 12:14:28 +02:00
Michael Peter Christen
a8167e6e5b
clean-up: removed unused methods in kelondro
2012-10-06 03:34:52 +02:00
Michael Peter Christen
8219a445f3
refactoring
2012-09-21 16:46:57 +02:00
Michael Peter Christen
00c1c777fa
refactoring
2012-09-21 15:48:16 +02:00
orbiter
563d584420
removed more dependencies in cora from kelondro
2012-09-21 11:02:36 +02:00
Michael Peter Christen
4d29f59a27
removed warnings
2012-09-10 07:15:52 +02:00
orbiter
2094df2e4e
- correct length computation for BStringObject (bugfix suggested by
...
apfelmaennchen)
- using ASCII for string conversion for Strings generated from Integer
2012-08-26 17:46:40 +02:00
Michael Peter Christen
1687737771
Abstraction of HandleMap and HandleSet
2012-07-27 12:13:53 +02:00
orbiter
0cbda0b2b8
- replaced all length() == 0 and size() == 0 with isEmpty()
...
- replaced some length() > 0 and size() > 0 with !isEmpty() - cannot be
done automatically
- implemented some isEmpty() methods
2012-07-10 22:59:03 +02:00
Michael Peter Christen
b0c408788b
made class methods static where possible
2012-07-05 12:38:41 +02:00
Michael Peter Christen
d3964253ae
- added @SuppressWarnings to unused servlet method parameters
...
- removed unnecessary casts
- removed unnecessary throw statements
2012-07-05 09:14:04 +02:00
Michael Peter Christen
ea10766bfd
cleaned unnecessary nested code
2012-07-05 08:44:39 +02:00
Michael Peter Christen
ce8d4b87d9
fixes for new eclipse 'Juno' warning 'Resource leak'.
2012-07-02 10:27:46 +02:00
Michael Peter Christen
b9d42fd9c8
using com.google.common.io.Files instead of homebrew methods
2012-06-22 11:39:17 +02:00
Michael Peter Christen
3b992e6b00
using utf8 String compression in Webstructure database
2012-06-09 11:00:33 +02:00
Michael Peter Christen
701b9a28a0
Merge branch 'master' of ssh://git@gitorious.org/yacy/rc1.git
...
Conflicts:
htroot/PerformanceMemory_p.java
2012-06-08 09:16:16 +02:00
Michael Peter Christen
10c9c17d51
fixed handlemap spread factor and null iterator handling
2012-06-08 09:13:41 +02:00
Michael Peter Christen
461a0ce052
removed warnings
2012-06-05 20:03:43 +02:00
Michael Peter Christen
e0d8643226
- performance hacks
...
- added log warnings in case that search processes run into time-out
situations
- better concurrency for Integer formatter (used a non-synchronized
formatter before)
- bugfix for search termination (a poison pill was missing)
- added timeout parameters for search (again) -> target is, that they
are never reached.
2012-06-04 15:37:39 +02:00
Michael Peter Christen
10da7335ea
performance hack: use a hash cache for all hashes that are computed by a
...
byte array. If this hash is used in a HashMap (which is very often the
case) then this hack eliminates a lot of re-computations of the same
hash.
2012-05-30 16:59:13 +02:00
Michael Peter Christen
c15fcde1c8
add-on to latest commit
2012-05-21 17:52:30 +02:00
Michael Peter Christen
cf47d94888
performance hack to parse numbers inside of substrings without actually
...
generating a substring. This avoids the allocation of a String object
ech time a substring is parsed. Should affect CPU load during RWI
transmission.
2012-05-21 13:40:46 +02:00
Roland 'Quix0r' Haeder
a093ccf5eb
Now used synchronization in all close() methods to make sure all objects
...
are 'closed' in an ordered way
Conflicts:
source/de/anomic/http/server/ChunkedInputStream.java
source/de/anomic/http/server/ChunkedOutputStream.java
source/de/anomic/http/server/ContentLengthInputStream.java
source/net/yacy/cora/protocol/Domains.java
source/net/yacy/cora/services/federated/solr/SolrShardingConnector.java
source/net/yacy/cora/services/federated/solr/SolrSingleConnector.java
source/net/yacy/document/content/dao/PhpBB3Dao.java
source/net/yacy/document/parser/html/AbstractTransformer.java
source/net/yacy/kelondro/blob/BEncodedHeap.java
source/net/yacy/kelondro/blob/HeapReader.java
source/net/yacy/kelondro/index/RAMIndexCluster.java
source/net/yacy/kelondro/io/ByteCountInputStream.java
source/net/yacy/kelondro/logging/ConsoleOutErrHandler.java
source/net/yacy/kelondro/table/SQLTable.java
2012-05-14 07:41:55 +02:00