reger
bb0076c3dd
fix: assure close inputstream in TranslatorXliff after reading xlf file
...
by using try-wiht-resource block
2016-05-29 01:25:47 +02:00
reger
6384b7d82e
fix NPE in Load_MediawikiWiki servlet in intranet mode
...
- in intranet mode getip returns null causing a NPE
- adjust starturl (which was set to http://localip/repository ) which is never the start url for the Mediawiki
+ correct javadoc for seed.getIP()
2016-05-27 03:10:25 +02:00
Michael Peter Christen
596b5dfa59
add the JRE version in the seed. Purpose: identify if it is possible to
...
migrate to new JRE version
2016-05-24 23:11:59 +02:00
reger
4cc38e979d
add InputStream close after reading input file (Vocabulary_p servlet)
2016-05-24 00:26:28 +02:00
reger
6bf9c55584
adjust Solr select servlet to lates bugfix for boostquery (bq param)
...
to split query into multiple parameter on line separator in input query.
e.g. split "crawldepth_i_0^10.0 \n crawldepth_i:1^5.0"
but allow "url_file_ext_s:jpg OR url_file_ext_s:png" to be unsplitted
2016-05-22 22:43:56 +02:00
Burkhard
9a18e2297b
Merge pull request #51 from JeremyRand/multiple-boost-query
...
Fix multiple boost queries
2016-05-22 22:24:04 +02:00
reger
f0d7b93372
make use and activate autodetect charset in Vocabulary input from file
...
+ revert mistake of empty cn.lng
2016-05-22 05:38:26 +02:00
JeremyRand
433217b33e
Properly support multiple Boost Queries. (Previous code was broken because it concatenated multiple Boost Queries together rather than passing Solr an array.)
2016-05-20 20:17:51 -05:00
JeremyRand
58824dfa6c
Refactor escaping in config file read/write code. Now it uses Apache Commons StringUtils instead of RegEx.
2016-05-20 20:17:51 -05:00
reger
9e94989237
upd to PDFBox 2.0.1
2016-05-20 23:12:16 +02:00
reger
d0a571bed2
del cytag trail for own index.html (save resource not used by default)
2016-05-19 01:59:00 +02:00
reger
de46879637
fix SeedDB.get(byte[]) hash string compare (for returning own seed shortcut)
2016-05-17 02:07:49 +02:00
reger
24b0fa2a38
extend snapshot Html2Image.pdf2image to use PDFBox image export capability
...
if no external tool installed (and for Win)
Resulting jpg are not always perfect (if graphic included) but imho sufficient.
2016-05-16 02:13:33 +02:00
reger
eb2a00b1d8
fix NPE on missing crawldepth_i
2016-05-15 01:26:38 +02:00
reger
efb9f1a8b7
save resource for unused blacklistFiles map
2016-05-12 00:13:57 +02:00
reger
5f113be760
cleanup connectPeer & yacyVersion.latestRelease usage
...
obsolete since
527b3decde
2016-05-06 21:05:15 +02:00
reger
7097dcbdbd
cleanup hack for partial Solr update on multivalued datefields
...
has been fixed in Solr http://issues.apache.org/jira/browse/SOLR-8050
2016-05-06 02:47:04 +02:00
reger
f10ea3c155
clean-out unused SwitchboardConstants
2016-05-05 00:55:22 +02:00
reger
ef24593347
delete obsolete SEARCHRESULT busythread constants
...
not used since 29.05.2013 18:27:27
0c1a018bbd
2016-05-04 01:30:10 +02:00
reger
125b5e26a5
apply bugfix for ChartPlotter from Pullreq 42
...
https://github.com/yacy/yacy_search_server/pull/42
thanks to otteresk (https://github.com/otteresk )
2016-05-03 03:06:06 +02:00
reger
06ce9ae711
prevent "unchecked conversion" compiler message
...
+ include "translate" property in xlf "trans-unit" export
2016-05-01 02:22:05 +02:00
reger
b4a576dbdf
exclude unused protocol param "duetime"
...
(receiver interpretes param "time" only)
2016-04-25 01:57:33 +02:00
reger
3bd6ae8d8b
keep addon/Notepad++ keyword marker on lng export
...
(length of remarks devider line)
+ harmonize status_p.inc lng text
2016-04-21 00:51:08 +02:00
reger
16837d60c7
fix version in locale version file
...
(it's compared to full version)
2016-04-17 22:54:28 +02:00
reger
0fb01e429e
fix migration, account for ssl port in config (for auto-disable https)
2016-04-17 04:42:05 +02:00
reger
7be1c7a05a
fix logger name
2016-04-17 03:20:14 +02:00
reger
1d940e5a94
upd commons-compress 1.11
2016-04-16 23:31:03 +02:00
reger
7789c32c82
delete crawl queue on init exception
...
(happens occasionally on path name vaiolation and will never get resolved)
2016-04-16 00:22:48 +02:00
reger
f781b9dd47
revert call condition f. migration.installSkins
...
(a bug introduced in fb8ae14b21
,
see comment on that commit )
2016-04-14 22:14:32 +02:00
reger
3adb670f44
remove never used Domains.myHostNames set
2016-04-14 02:54:41 +02:00
reger
6ecc180299
fix rwi doubledom return best (highest) ranking
2016-04-12 03:55:43 +02:00
reger
2343e3f1cd
keep and update existing xlf translation master instead of create new
...
in utility CreateTranslationMasters
+ small fixes in lng's
2016-04-09 23:25:05 +02:00
reger
a1935f485f
Added utility class CreateTranslationMasters to create a language independant
...
translation master as source to harmonize individual translation files
Included a main to create masters in YaCy an xliff format for testing
+ restrict TranslatorXliff to use only entries with State=translated
P.S. used https://open-language-tools.java.net/editor/about-xliff-editor.html to
experiement with xlf output (haven't a Pootle avail.)
2016-04-05 01:57:32 +02:00
reger
acaf51b296
keep ConfigLanguage_p as 1st entry in exported translation file
...
+ rem untranslated text & some typo fixes in several translations
(considering to create a translation master file to harmonize entries)
2016-04-04 02:56:19 +02:00
reger
61c5b6b403
fix empty drop down list in ConfigLanguage after wrong/empty download
...
+ add xliff translated attribut
+ append japanese lng name
2016-03-31 01:51:25 +02:00
reger
4eddabee42
translate Network History screen -> de
...
+ remove leftover debug line
2016-03-30 01:09:13 +02:00
reger
90c79014ae
remove unused translator routine which also doesn't handle rel path input
...
+ correct some language file match issues
2016-03-29 21:31:02 +02:00
reger
902e79e261
Introduce a TranslatorXliff wich can read/write xliff from/to internal translation map.
...
This eases up suggested initatives from http://mantis.tokeek.de/view.php?id=649
Allows longer term also to store translation maps for the htroot files
in standardized/reuseable xliff format ( http://docs.oasis-open.org/xliff/xliff-core/xliff-core.html ).
+ added test case creating and comparing xliff file with internal custom prop file.
(currently the introduced class is not used in core code)
2016-03-28 23:26:30 +02:00
reger
d9adc2c255
load handler for Transparent Proxy on startup only if feature is activated
...
to save the resources and keep handler chain small if the feature is not used.
+add a warning message on settingsack_p page to restart on first activation
2016-03-25 05:26:48 +01:00
reger
ec24a0c85a
add test case for optimized toTokens()
2016-03-24 19:26:38 +01:00
reger
cada24f918
adjust utility ListNonTranslatedFiles for path compare on windows
...
(backslash replace)
2016-03-20 23:46:39 +01:00
reger
fb8ae14b21
make migration version safe
2016-03-20 03:34:28 +01:00
reger
258cd41577
reduce logging (EmbeddedSolrConnector.query)
...
mainly to reduce the frequent metadat checks like
> EmbeddedSolrConnector.query QUERY: q={!cache=false raw f=id}xXxXxX&rows=1&start=0&fl=id,load_date_dt
(p.s. direct servlet queries logged via AccessTracker.addToDump)
2016-03-14 22:32:06 +01:00
reger
6783ef5540
move example code SearchClient out of yacycore package
...
to example directory
2016-03-14 02:22:06 +01:00
Michael Peter Christen
b89465d952
0N - basic dump upload servlet infrastructure, to share index dumps
...
within an experimental new sharing model
2016-03-11 18:12:13 +01:00
Michael Peter Christen
f12a900f3e
harmonization of http post of files for one and several files - this had
...
been differently - and wrong for several files. also: base64-encoding
for gzipped push files because our data structures currently only
supports ASCII POST pushes..
2016-03-11 08:59:33 +01:00
Michael Peter Christen
849ab671a9
0n: modified the p2p bootstraping process - rules had been too tight and
...
did not support the re-start of a network with just one principal peer.
2016-03-11 08:54:42 +01:00
reger
764f5100f0
fix delete of temp file after odt % ooxml parser
...
Close zipfile after parsing
2016-03-04 23:05:55 +01:00
reger
379e9b330d
use supplied url port to get robots.txt in crawlers hostqueue
2016-03-02 00:12:34 +01:00
reger
58a959403d
fix mixed logfactory in UrlProxyServlet,
...
Class doesn't use functions of declared ancestor, change to extend on httpservlet
2016-02-27 03:44:43 +01:00
Michael Peter Christen
2494a820c7
0N - added recording of dump exports if given time frame is not negative
2016-02-24 15:13:20 +01:00
Michael Peter Christen
ef2cc4f690
Merge branch 'master' of git@github.com:yacy/yacy_search_server.git
2016-02-24 11:19:32 +01:00
Michael Peter Christen
a6bf0b1649
0N - added option to generate index export files for a specific number
...
of minutes in the past and reverted latest change. The export file dump
will now contain four data elements: f - first date of index entry write
date, l - last date of index write date, n - now-date of index dump
time, c - count of numbers inside the dump. '0N' denotes a series of
changes which will lead to the opportunity to exchange index data dumps
in a way that is needed to integrate ZeroNet index data. This will be
based on index dump sharing; that causes this commit.
2016-02-23 18:56:20 +01:00
reger
6d56beaed8
fix assertion exception in toString of MultiProtocolURL
...
toString of AnchorURL and MultiProtocolURL are identical code
(no need to override or to protect call to parent)
as reported in https://github.com/yacy/yacy_search_server/issues/43
2016-02-21 00:23:00 +01:00
reger
42a7bdb2af
fix SolrSelectServlet authentication to default to true
2016-02-20 22:30:15 +01:00
reger
dbb28bb4f3
del unused statistic parameter (from status servlet)
2016-02-17 22:47:03 +01:00
reger
06d0e2aeb9
result heuristic (also used in greedy learning mode) to use outbound links if result is full index doc. Otherwise use default loader methode.
...
- Above brought up that parser start url parameter, declared as AnchorURL uses only methodes of parent object DigestURL (changed parameter declaration accordingly).
2016-02-16 02:05:58 +01:00
reger
caf9e98f09
put metadata dc_publisher in corresponding schema field
2016-02-14 21:13:25 +01:00
reger
38e2b054d4
remove servlet classloder internal cache map (to save the resources, cache hits marginal)
...
- DefaultServlet includes already a class cache "templateMethodCache" which is emptied
on low mem status
- avoid classloader cache gets has no hits but over time holds all (used) servlet classes
2016-02-12 01:20:03 +01:00
reger
6f0b073bf3
override detected language (statistic langdetect) only with TLD determided
...
language if langdetect probability is not high.
+ additionally truncate zh-cn / zh-tw returned by langdetect to 2 char ISO639-1 zh
used by YaCy
2016-02-07 21:16:22 +01:00
reger
b65e2b527d
include use of condenser's content text for language detection.
...
Language identification may show poor performance on documents with short or no
title but clear lang indication in text content. Using content text too
improves lang detection.
+ remove double caching of text in Identificator
2016-02-07 01:52:32 +01:00
reger
937fbb0b9f
correct isHidden() for smb from last commit
2016-02-04 19:20:27 +01:00
reger
535d4bf75f
respect hidden attribute for file and smb directory listing
...
(hidden directories are not listed, effects crawling of local file system)
2016-02-04 19:16:00 +01:00
reger
c28142095a
add findClass() to servlet class loader (used in YaCyDefaltServlet)
...
In the 2 cases where servlet calls servlet the jvm classloader chain is
invoked and servlet class loaded by jvm loader (successful while requiring
htroot in system classpath). This patch uses the standard override design
for loaders to handle these cases (making in not longer crucial to have htroot
in system classpath, as this classLoader is mainly used for servlets and
looks in this case for the class in the configured path).
+ As the default classloader is parallelcapable we should register this too.
2016-02-02 03:44:01 +01:00
reger
a6617ad887
expand initRemoteCrawler() to terminate worker threads if called to deactivate
...
remote crawl.
On startup we save the resources for remote crawler if disabled. Once started
threads are running idle after disable remote crawl. Now threads are terminated
to save the resources also while disabeling during runtime.
+ remove empty class Channels
2016-01-28 23:14:09 +01:00
reger
2048b7e057
support scraping start-/enddate from html tag with property "datetime"
...
This may be used in html5 <time> tag (which we don't explicite support yet for date in content scraping).
2016-01-26 21:27:44 +01:00
reger
900d4584ba
complet resource cleanup of lists in contentscraper's close()
2016-01-25 23:54:20 +01:00
reger
1f18653de0
pass parsed swf content trough htmlscraper
...
Swf may contain subset of html tags which shoul'd appear as text.
Especially <font> tag may totally screw up metadata servlet if not filtered out.
2016-01-21 02:55:05 +01:00
reger
18ecf57792
add support of compressed swf to swfParser
...
from JavaSWF2 (source compatible to WebCat).
Moved swf file signature check to parser
Changed use of synced vector to list swf InStream
2016-01-20 00:58:29 +01:00
sixcooler
5cb7ba0dc4
fix for connections not getting closed to get favicon.ico during seach
2016-01-19 20:57:22 +01:00
reger
ed3e16e092
apply remote result count config value to Bookmark Autosearch
...
+ prepare to make the widely unused Bookmark feature optional
2016-01-15 02:10:10 +01:00
Ryszard Goń
a98c395023
Add the Autocrawl thread
2016-01-14 00:50:23 +01:00
Ryszard Goń
1728cd30c6
Create autocrawl profiles
2016-01-12 16:28:34 +01:00
reger
ff27824964
fix swfParser reading file signature
...
before passing to library (current version expects data w/o signature)
2016-01-10 01:16:31 +01:00
reger
c91e712178
further refactor using standard java / (one) utf-8 charset variable
...
extending initiative of commit 9a25751850
2016-01-07 16:17:37 +01:00
luc
571bc55937
Refactoring : use StandardCharsets constants instead of hard-coded
...
charset names.
2016-01-05 23:37:05 +01:00
reger
1af0e9ef74
remove workaround for Solr bug regarding multivalued date fields
...
fixed in 5.4.0
http://issues.apache.org/jira/browse/SOLR-8050
2016-01-03 01:11:27 +01:00
sixcooler
5a35f9383a
bump to solr/lucene 5.4.0
2016-01-02 21:07:50 +01:00
reger
a58d34a4e8
check error URL cache before adding errorDoc to index
...
- del obsolete related switchboardconstant
2016-01-02 05:03:57 +01:00
reger
e9539b1086
reintroduce special handling of file upload multipart/form-data from HTTPDemon.parseMultipart
...
- add filename to parameter fieldname
- add filecontent to special parameter fieldname$file
(some servlets use this $file parameter)
fix for http://mantis.tokeek.de/view.php?id=542
2015-12-31 03:04:13 +01:00
reger
cd26717ba2
fix low memory status hint (dht-in disabled)
...
http://mantis.tokeek.de/view.php?id=619
2015-12-29 20:38:45 +01:00
reger
a5faf73afa
remove obsolete yacy.init entries interaction.*
...
(related to removed triplestore)
2015-12-29 15:41:19 +01:00
sixcooler
dce1cb65c4
Merge remote-tracking branch 'choose_remote_name/master'
2015-12-28 23:20:42 +01:00
reger
46ac0867ff
fix poison mediawikiimporter output queue also after ExecutionException
...
in worker thread.
Writer of importer keeps needs a poison to close the file. On exception (e.g. OOM)
add a poison marker in outer most try/catch to assure output queue will terminate
in this condition too (and closes+renames the surrogate/in/xxx.prt file)
2015-12-28 02:32:00 +01:00
reger
a7591d3ed0
fix mediawikiimporter number format exception on coordinate parsing
...
handle uncomplete metadata like "NS=43/50//N".
For other {expr ... } type entries a try catch added
2015-12-27 01:59:15 +01:00
reger
9da1712a31
increase http header EXPIRES for css and images in DefaultServlet
...
to increase browser cache hits for not changing content
2015-12-26 17:35:46 +01:00
reger
6d54eb3d36
skip loading document on crawl start for YMark bookmarks
...
by adding a constructor giving the already loaded document as parameter.
2015-12-26 01:15:07 +01:00
reger
80e2c82249
fix NPE on empty blog importfile parameter
2015-12-24 02:00:45 +01:00
reger
e84d94f8ca
fix mime table for ms office / open office documents
...
(causing wrong parser detect in intranet mode)
2015-12-22 17:48:24 +01:00
reger
45b9bd8403
adjust MultiProtocolURL.protocol detection to handle mailto with "://" in parameters,
...
and feeding hyperlinks to webgraph processing.
2015-12-21 04:42:26 +01:00
reger
d5fd031449
fix reading of ippattern config array in URLProxy
2015-12-20 15:51:54 +01:00
reger
b7e8358645
make use of header.getContentType where possible (mime is normalized afterwards)
...
otherwise use header.mime() differentiated in prev. commit.
2015-12-20 15:49:24 +01:00
reger
7a8c077838
fix HeaderFramework.mime() to strip charset parameter.
...
Differentiate mime() and getContentType() which gives the raw header field.
This improves parser detection if charsets are included in http content-type field.
2015-12-20 06:44:16 +01:00
reger
b4b6910d60
fix (todo): correct doc.id of remote search result if no match with newly
...
calculated doc hash if different.
Testing showed that in some cases delivered url doesn't match the local
calculated hash. In this case replace doc.id (and host_id_s) with calculation
from url.
2015-12-20 02:10:49 +01:00
reger
dec3e6ad96
fix: adjust urlstub for mailto links
...
(skip protocol)
2015-12-19 20:13:33 +01:00
reger
cb83e65f89
drop returning document language "en" if unknown (fix todo)
...
which also harmonizes handling of query.modifier for rwi and solr results
(to result must match a given language filter)
2015-12-19 01:42:35 +01:00
reger
0c5548a7ff
fix (todo) remove redundant holding of email link nameproperty in parser document
2015-12-18 02:35:44 +01:00
reger
71c416f383
show mailto links in ViewFile.html linklist
2015-12-18 01:11:55 +01:00
reger
6b7c10cef8
fix dc:date in mediawikiimporter/document.writexml to use lastmodified
2015-12-17 02:53:10 +01:00
reger
14803d58cd
let html scraper accept html5 <link rel="icon"> for favicon links
2015-12-17 00:36:08 +01:00
luc
b4cdacee76
Merge branch 'master' of https://github.com/yacy/yacy_search_server
2015-12-16 03:26:06 +01:00
luc
ba0a293f5c
Corrected another case of
...
org.apache.lucene.store.AlreadyClosedException" occuring when
SearchEvent.cleanup() was called while committing local solr index.
2015-12-16 03:25:07 +01:00
reger
4d2b934487
prevent mailto links getting into parser result document's in/outbound link collection
...
by checking mailto scheme early.
- fix upper case mailto protocol assignment
- add test case for getProtocol
2015-12-16 03:01:17 +01:00
luc
8c4ab9c76b
Added an option to eventually limit size of remote solr documents put to
...
local index. See mantis #626 .
2015-12-16 02:20:03 +01:00
luc
a2c08402af
Merge branch 'master' of https://github.com/yacy/yacy_search_server
2015-12-15 23:30:30 +01:00
luc
70595d05d0
Modified MemoryControl.main() test to properly end for better results
...
displaying.
2015-12-14 23:49:28 +01:00
sixcooler
1be67d9ab6
CachedSolrConnector was replaced by ConcurrentUpdateSolrConnector years
...
ago - time to let it go
Commented out unused table of cache-objects
2015-12-14 21:33:27 +01:00
reger
28b8bc290a
fix use of NETWORK_SEARCHVERIFY for rwi verification
...
was not used to set the searchevent parameter (done in SearchEventCache.getEvent)
- remove unused corresponding QueryParams.filterfailurls param.
2015-12-13 20:01:49 +01:00
reger
020630efd8
remove unused network scanner parameter from queryparameter
...
Search event is not using networkscanner
(removed filterscannerfail param always init to false)
2015-12-13 02:50:08 +01:00
luc
ad5586f8f6
Merge branch 'master' of https://github.com/yacy/yacy_search_server
2015-12-08 03:35:36 +01:00
luc
8ebefa4233
Fixed MediaWiki import : DCEntry conversion to SolrInputDocument was
...
failing. Looks like it was broken since Commit
b43811d38c
2015-12-08 03:34:03 +01:00
luc
7736ee5a42
Updated MediaWimporter main() : display usage in console and stop
...
properly without calling System.exit
2015-12-08 03:30:51 +01:00
reger
cdb8f3b10d
make current ranking score value avail. to search interface / api
...
Update the result score result field with the result queue ranking value to reflect
the actual calculated/used score,
for rwi & solr stack results.
(calc. etc. is unchanged, it's just that result entry carries the latest val
as api retrieves the number from it)
2015-12-08 03:17:32 +01:00
luc
27d11f8671
Fixed isSolrDump function : PushBackInputStream was not unread when
...
returning false (for example with a WikiMedia dump).
2015-12-07 21:58:36 +01:00
Michael Peter Christen
135a123a77
less logging in new language detection
2015-12-03 00:39:15 +01:00
Michael Peter Christen
ef8cd80593
fix for npe
2015-12-03 00:33:13 +01:00
reger
0347bfa71f
Apply collection query constraint/modifiert to rwi result stack.
...
Collection is not available in pure rwi entries (but in local solr metadata)
But if user wishes to filter by query constraint also rwi shall adhere to this
(even if only rwi entries with parsed or solr received metadata may fit)
2015-12-02 22:57:59 +01:00
luc
2a67d2ba6f
Corrected error management for unsupported image formats, parsing
...
errors, and unavailable resources : avoid logging to much Exceptions as
these errors easily occur when searching images.
2015-12-01 01:06:01 +01:00
Michael Peter Christen
d6e9834040
Merge branch 'master' of
...
https://github.com/Scarfmonster/yacy_search_server
# Conflicts:
# .classpath
# build.xml
2015-11-30 16:54:54 +01:00
Michael Peter Christen
d82d311995
Merge branch 'master' of https://github.com/luccioman/yacy_search_server
...
# Conflicts:
# .classpath
2015-11-30 13:34:10 +01:00
reger
b5371ea8c1
read/init crawl queue in a thread
...
to speed-up YaCy start on large existing crawler queues
2015-11-29 05:19:39 +01:00
reger
1160b13172
remove unused md5 from ViewFile servlet params
2015-11-28 23:09:15 +01:00
reger
e163ea88f6
fix vsdParser (Visio) parser return statement
...
(final block un-necessary throw)
2015-11-28 02:43:38 +01:00
reger
b2c8bc0ae6
remove md5_s from default index fields
...
it is not assigned a value / not used
Due to above also excluded from transfer protocol.
2015-11-27 02:41:02 +01:00
luc
e40ae0943b
- No max dimensions specified : render raw image data when source and
...
target image format are the same.
- Corrected scaling condition.
2015-11-26 09:30:43 +01:00
reger
90686a75a2
fix flux factor (additional crawl delay by access count) calculation
2015-11-25 01:34:41 +01:00
luc
4af27289e5
Merge branch 'master' of https://github.com/yacy/yacy_search_server
2015-11-23 09:01:25 +01:00
reger
297fdb60d3
throw exception if crawler hostqueue can't create hostpath directory.
...
In rare cases hostname may not be a valid filesystem directory name,
which can't be created (e.g. containing '*' char). To prevent crawl queue
looping on this invalid entry by throwing a malformedurlexception.
2015-11-22 21:26:18 +01:00
luc
755efac17d
Use same max file size when loading all resource bytes or opening stream
...
content
2015-11-20 19:35:39 +01:00
luc
bc6c79fc12
Corrected scaling function for non RGB images.
2015-11-20 14:35:36 +01:00
luc
1565559df8
Refactoring : extracted write InputStream method.
2015-11-20 09:42:24 +01:00
luc
f0478bb14d
BMP and ICO image formats support : integrated /haraldk/TwelveMonkeys
...
imageio-bmp-3.2 library.
- better BMP format flavours support
- handle PNG encoded icons
- handle transparency
Added some javadoc url references to .classpath
2015-11-20 09:38:16 +01:00
luc
07437986e7
Merge branch 'master' of https://github.com/yacy/yacy_search_server
2015-11-20 08:15:24 +01:00
reger
97cc03ef6a
start using a template for urlproxy header
...
It is included as iframe /proxmsg/urlproxyheader.html
to allow full servlet functionallity and flexibility to display some
index/meta data in future.
2015-11-20 01:49:56 +01:00
luc
f01d49c37a
Process large or local file images dealing directly with content
...
InputStream.
2015-11-18 10:15:38 +01:00
luc
3c4c77099d
If available, check content length before downloading. Check also
...
content length is not over Integer.MAX_VALUE.
2015-11-18 10:11:38 +01:00
luc
5bbb2e1730
Ensure resource is closed when reading a full file InputStream
2015-11-18 10:08:06 +01:00
luc
6291a57300
Merge branch 'master' of https://github.com/yacy/yacy_search_server
2015-11-18 08:49:31 +01:00
reger
0d3c5b223e
have psParser cleanup temp file
2015-11-17 23:45:29 +01:00
reger
7d0d19cb8e
avoid File.deleteOnExit() on temp files
...
JVM registers each file in a list regardless of already deleted and never
cleans up the list during runtime.
This accumulates to a considerable amount of mem during large crawls and/or
long uptime.
To tackle this, all temp files are now created in a subdir of java.io.tmpdir
and the jvm tmpdir property is set to this subdir, which is deleted by
code on shutdown.
Additionally let pdfParser use this tmp subdir too.
2015-11-17 22:27:07 +01:00
luc
bfe51001e3
Merge branch 'master' of https://github.com/yacy/yacy_search_server
2015-11-17 08:30:32 +01:00
reger
02e4489a23
set tmpfile.deleteOnExit by default,
...
to make sure files are removed on shutdown.
2015-11-16 21:37:45 +01:00
reger
2985baaa01
Exclude repetitive protocol part in tokenized url
...
used as description if none is avail. from parser.
2015-11-16 01:06:20 +01:00
reger
ca3d26a401
harmonize wordsintitle & CollectionSchema.title_words_val calculation,
...
remove obsolete partial init of wordreference from urimetadata
2015-11-15 06:06:37 +01:00
reger
52a9040ae6
Sort out double keywords (dc_subject) early in parsed documents
...
- by direct using Set vs. List
- remove not neede String[] getter
2015-11-13 01:48:28 +01:00
luc
49331dc523
Merge branch 'master' of https://github.com/yacy/yacy_search_server
2015-11-12 08:21:56 +01:00
reger
47d70732f6
improve locale translator
...
- skip empty line
- robustness file section detection (space independant)
2015-11-11 00:57:51 +01:00
sixcooler
646afe9183
do not store subfield *_coordinate + make all num-fields being docvalues
2015-11-10 20:45:33 +01:00
sixcooler
194df613de
not using 'location' as defaultfacetfield - since we removed it being
...
default.
2015-11-10 20:43:58 +01:00
sixcooler
d3b9349b6f
simplification / speedup of GenerationMemoryStrategy
2015-11-10 20:39:46 +01:00