Commit Graph

9271 Commits

Author SHA1 Message Date
Michael Peter Christen
592adf7ccb fix for domain navigation 2013-02-02 07:21:18 +01:00
Michael Peter Christen
4ca1b76627 less search overhead when first result set is smaller than requested 2013-02-02 07:20:56 +01:00
Michael Peter Christen
f748b0aa7c NPE fix 2013-02-02 07:20:02 +01:00
Michael Peter Christen
7dfcc92b71 Merge branch 'master' of ssh://git@gitorious.org/yacy/rc1.git 2013-01-31 13:15:42 +01:00
Michael Peter Christen
0b6566a389 optimizations when starting large crawl requests with many start urls in
one request:
- allow larger match-fields in html interface
- delete all host hashes at once from zurl
- when deleting by host, do not count size of deleted entries since that
was the reason it took so long
2013-01-31 13:15:28 +01:00
orbiter
a2160054d7 ability to create vocabularies also without any objectspace: this
iterates over all urls in the index do create terms
2013-01-30 19:33:48 +01:00
orbiter
ecc10a752c fixes to index enumeration for vocabulary production 2013-01-29 18:14:14 +01:00
reger
be5d3a1066 adding classpath to Manfiest of yacycore.jar
- this allows to start w/o giving explicite java -cp (just java -jar lib/yacycore.jar works)
- especially helpful while running YaCy as Win service, 
  making it obsolete to adjust classpath cfg of the service wrapper on upgrades of lib/*.jar's
2013-01-29 03:01:57 +01:00
Michael Peter Christen
be27567b53 allow more links when starting a crawl by file 2013-01-28 17:50:23 +01:00
reger
3777b338c7 bugfix: location url for migrate urldb button onclick 2013-01-27 06:13:49 +01:00
reger
8447814a31 correct headermenue in migrateurldb_p.html
- update NetBeans project path
2013-01-26 23:43:09 +01:00
Michael Peter Christen
99185d7048 one more fix for author_sxt 2013-01-26 03:59:39 +01:00
Michael Peter Christen
b6ae6262f6 - add the copyField author_sxt only if author exists
- set the solr default search field according to existing fields
2013-01-26 03:34:46 +01:00
Michael Peter Christen
088373b4ea catch exception if solr connection change fails 2013-01-25 16:06:58 +01:00
Michael Peter Christen
8a55fd96e9 Merge remote-tracking branch 'aleksejs/rutrans3' 2013-01-25 13:57:10 +01:00
Aleksej
3252223055 Russian translation fixes and additions 2013-01-25 16:05:48 +04:00
Dmitriy Kazimirov
7293c0413b Russian localization:index.html fix 2013-01-25 10:20:21 +01:00
sixcooler
3a13906121 clear some more caches if running out of memory 2013-01-25 04:24:36 +01:00
Marc Nause
6fc4bdddbd *) fixed admin password configuration 2013-01-24 20:09:33 +01:00
Michael Peter Christen
e23a596c1d added a copyField for author_sxt for automated schema generation 2013-01-24 18:25:28 +01:00
Michael Peter Christen
8651ec35fe turned author_s into the multi-valued field author_sxt 2013-01-24 18:24:31 +01:00
Michael Peter Christen
f1a4feda3e security fix for suggest (don't let users ask for too much) 2013-01-24 17:57:28 +01:00
Michael Peter Christen
244b157299 fix for external solr schema definition 2013-01-24 16:34:15 +01:00
Michael Peter Christen
4589afe056 fix NPE when solr does not deliver snippets 2013-01-24 14:12:31 +01:00
Michael Peter Christen
0fe7b6fd3b migrated the index export methods from the old metadata to solr. Now
exports are done using solr queries. removed superfluous methods and
servlets.
2013-01-24 12:39:19 +01:00
Michael Peter Christen
1768c82010 removed field selection because that created documents with that field
only which was not useful when re-writing the same document
2013-01-24 03:26:38 +01:00
Michael Peter Christen
8eebeea533 fix for search result link in ViewFile 2013-01-24 01:50:59 +01:00
Dmitriy Kazimirov
5bed1a7893 Russian localization update 2013-01-23 14:43:00 +01:00
Michael Peter Christen
31e854bef6 Merge remote-tracking branch 'copro/master' 2013-01-23 14:41:17 +01:00
Michael Peter Christen
4735bd47f4 - changed solr commit call and added an optimize option. Since Solr
4.0.0 there is a new softcommit feature which implements a
near-real-time (NRT) search option. The softcommit does not do IO and
does not cause performance issues.
YaCy has now an extension in its solr connectors to use the softcommit
feature. The softcommit call now replaces all places where a hard commit
was used. Furthermore the commit strategy in when doing a search from
the web interface was changed (it's done every time before a search is
done).

The softcommit feature was implemented because it was needed for the
following changes (customer demands), which is also included in this
git commit:

- added a feature to identify all documents which have unique titles
and/or unique descriptions. These unique flags are disabled by default.
- added also a feature to set a flag when the url from a canonical tag
is equal to the document url. This is also disabled by default.

To support the new softcommit strategy, the commitWithinMs option was
set to -1 do disable automatic commit based on document insert times. If
documents are inserted permanently then also a commit would happen
permanently whenever the commitWithinMs time is reached. This would
conflict with the regular autocommit of 10 minutes and the new
softcommit strategy.
2013-01-23 14:40:58 +01:00
Copro
0025983993 Fix typo embedd -> embed 2013-01-23 04:11:55 +01:00
Copro
3ea8380959 Adding Vimeo tag to wiki commands to embedd Video video with id 2013-01-23 04:00:15 +01:00
Copro
ee9d7fd93d Added feature to embedd Youtube videos to wiki commands for usage in
Wiki, Blog or other servlets
2013-01-23 02:43:58 +01:00
Michael Peter Christen
ec927ea72b Merge remote-tracking branch 'reger/master' 2013-01-22 17:01:49 +01:00
Michael Peter Christen
7159ed2a7d Merge remote-tracking branch 'copro/master' 2013-01-22 17:01:18 +01:00
Copro
946fad48c7 Some more German translation reducing the amount of Unused String
messages
2013-01-22 15:33:49 +01:00
Aleksej
6690dac845 Russian translation fixes not merged due to conflict 2013-01-22 16:19:07 +04:00
Michael Peter Christen
9ccdd21d76 Merge remote-tracking branch 'aleksejs/fixtrans'
Conflicts:
	locales/ru.lng
	
Tried to merge this but I had to made this 'blind'.
Sorry if I deleted something that was right.
2013-01-22 11:54:38 +01:00
Copro
de7c3d95b4 Added German translation for HostBrowser.html 2013-01-22 05:14:37 +01:00
Dmitriy Kazimirov
5e5ae01909 updated Russian localization for update system 2013-01-21 18:07:18 +01:00
Dmitriy Kazimirov
f9c65078f0 A little more fixes for Russian localization 2013-01-21 18:07:08 +01:00
Dmitriy Kazimirov
ca01d225db A little more fixes for Russian localization 2013-01-21 18:07:00 +01:00
Dmitriy Kazimirov
9dc0bea1dc Little more correct and readable Russian localization 2013-01-21 18:06:51 +01:00
Dmitriy Kazimirov
c1b9113a68 Little more correct and readable Russian localization 2013-01-21 18:06:43 +01:00
Dmitriy Kazimirov
9cc72df176 More Russian translations. And if some text is not translated it will be in English and not German 2013-01-21 18:06:02 +01:00
Michael Peter Christen
db024a4e19 added new solr fields (unused yet; implementation will follow) 2013-01-21 18:02:29 +01:00
Michael Peter Christen
f5fd2aea18 removed archaic migration code 2013-01-21 17:59:42 +01:00
Michael Peter Christen
9b5bdae1b4 Reverted setting of MMapDirectoryFactory from solrconfig; see
http://forum.yacy-websuche.de/viewtopic.php?p=27509#p27509
Instead, in the start script is checked if the host is a 64 host and
-Dsolr.directoryFactory=solr.MMapDirectoryFactory is set as java option

Reverted the ramBufferSizeMB setting (this was not enabled anyway)
because that may be too much memory for small peers and embedded
systems.

Activated the mergeFactor 4; this was commented out by mistake
2013-01-21 17:55:28 +01:00
reger
f8f7f33596 add Maven build script 2013-01-20 21:08:59 +01:00
orbiter
eb68a30947 solr performance settings
the target of these performance settings is the reduction of IO in
general and during search in particual.
- reduced mergeFactor to 4. This will increase the IO during indexing,
but will reduce IO during search. It will also greatly reduce the number
of open files which should make it possible to have overall larger
indexes until the number of open files in an OS is reached.
- increased ramBufferSizeMB to 256mb. This will reduce the number of
commits. This change may compensate the reduction of the mergeFactor.
- disabled updateLog. This is a real-time search feature which is
available in YaCy anyway because a commit is forced if index.html is
called. The updateLog feature causes a lot of IO during indexing and
search and produced a lot of files in SEGMENTS/solr_40/data/tlog
2013-01-19 11:21:33 +01:00