Marown/yacy_search_server
1
0
Fork 0
mirror of https://github.com/yacy/yacy_search_server.git synced 2024-09-22 00:00:59 +02:00
Code Issues Actions Packages Projects Releases Wiki Activity
8,682 Commits 1 Branch 20 Tags 836 MiB
24d9db1613
Commit Graph

51 Commits

Author SHA1 Message Date
Michael Peter Christen
24d9db1613 snippet retrieval loading processes may use a smaller minimum load time 
value than crawling processes. This speeds up the search result
preparation dramatically.
 2012-07-30 10:38:23 +02:00
Michael Peter Christen
1687737771 Abstraction of HandleMap and HandleSet 2012-07-27 12:13:53 +02:00
orbiter
69e743d9e3 - more abstraction for the RWI index as preparation for solr integration 
- added options in search index to switch parts of the index on or off
 2012-07-22 13:18:45 +02:00
Michael Peter Christen
5bd3c90907 - removed unnecessary semicolons 
- added default case for switch
 2012-07-05 11:18:31 +02:00
Michael Peter Christen
241dd8410a removed snippet pattern filter - it was not used 2012-07-05 09:21:27 +02:00
Michael Peter Christen
d3964253ae - added @SuppressWarnings to unused servlet method parameters 
- removed unnecessary casts
- removed unnecessary throw statements
 2012-07-05 09:14:04 +02:00
Michael Peter Christen
ea10766bfd cleaned unnecessary nested code 2012-07-05 08:44:39 +02:00
orbiter
78fc3cf8f8 refactoring and new usage of SentenceReader: this class appeared as one 
of the major CPU users during snippet verification. The class was not
efficient for two reasons:
- it used a too complex input stream; generated from sources and UTF8
byte-conversions. The BufferedReader applied a strong overhead.
- to feed data into the SentenceReader, multiple toString/getBytes had
been applied until a buffered Reader from an input stream was possible.
These superfluous conversions had been removed.
- the best source for the Sentence Reader is a String. Therefore the
production of Strings had been forced inside the Document class.
 2012-07-04 21:15:10 +02:00
Michael Peter Christen
1825f165b8 better integration of blacklist according to use case 2012-07-02 13:57:29 +02:00
Michael Peter Christen
03280fb161 removed segments-concept and the Segments class: 
the segments had been there to create a tenant-infrastructure but were
never be used since that was all much too complex. There will be a
replacement using a solr navigation using a segment field in the search
index.
 2012-06-28 14:27:29 +02:00
Michael Peter Christen
e0d8643226 - performance hacks 
- added log warnings in case that search processes run into time-out
situations
- better concurrency for Integer formatter (used a non-synchronized
formatter before)
- bugfix for search termination (a poison pill was missing)
- added timeout parameters for search (again) -> target is, that they
are never reached.
 2012-06-04 15:37:39 +02:00
Michael Peter Christen
8b974905ee changed log-in text for all servlets with authentication: 
- added hint how to set the password using a shell script
- added a shell script to change the password
 2012-05-24 13:24:31 +02:00
Michael Peter Christen
f294f2e295 bugfix to http://bugs.yacy.net/view.php?id=181 
tried to make a bit less 'noise' to dns server

also included: less processes in snippet fetch to reduce load during
search on small computers
 2012-05-19 01:06:33 +02:00
Michael Peter Christen
3e1bc9477f Merge branch 'master' of ssh://git@gitorious.org/yacy/rc1.git 2012-05-17 13:58:09 +02:00
Michael Peter Christen
5b3acc12cd Pattern.quote() replaces \\Q and \\E according to publication in 
http://www.cs.washington.edu/homes/mernst/pubs/regex-types-ftfjp2012.pdf
 2012-05-17 03:55:10 +02:00
Michael Peter Christen
89142d1e8d removed (not all) warnings 2012-05-16 13:42:32 +02:00
Michael Christen
9e5894c784 Removed handling of components objects for URIMetadataRows. 
This is a preparation to replace this rows with nodes from the node
store.
 2011-12-17 01:27:08 +01:00
admin
23afee58fe Merge branch 'master' of git://github.com/f1ori/yacy 2011-12-07 01:01:04 +01:00
apfelmaennchen
ff19fcdb28 bugfix for YMarks XBEL import and export; thanks to Dominic 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8138 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-12-06 23:17:57 +00:00
Michael Christen
9cd469e6d6 added pull request from als plus an NPE fix 2011-12-04 12:15:03 +01:00
apfelmaennchen
70bcfc150a - small bug fix to ymarks html importer 
- import of delicious.com exports has successfully been tested

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8132 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-12-02 23:24:40 +00:00
apfelmaennchen
b5d9f631e3 git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8128 6c8d7289-2bf4-0310-a012-ef5d649a1542 2011-12-01 22:25:23 +00:00
Al Sutton
8993cac4d8 Initial performance improvements 2011-11-30 11:15:54 +00:00
apfelmaennchen
77a080ced9 smaller fixes for YMarks 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8105 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-11-25 17:33:03 +00:00
apfelmaennchen
dd1482aaf5 further update to YMarks 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8100 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-11-25 00:32:18 +00:00
orbiter
c584db991f creating a bookmark from the search results now works again .. with new YMarks 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8092 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-11-24 14:57:09 +00:00
apfelmaennchen
564374d1fe - included YMarks in addition to old bookmarks in yacysearchitem.html; don't get confused by the old bookmark dialog, the ymark is automatically added silently beforehand. 
- reworked bookmark creation on crawlstart
- many smaller adjustments to ymarks


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8072 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-11-22 23:50:49 +00:00
apfelmaennchen
6287c2b4a9 YMarks: 
- introduced tag manager - a quite powerful tool (still not 100% stable, so be careful)

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8060 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-11-20 22:42:15 +00:00
apfelmaennchen
5581be12fb YMarks: 
- added backend and api for tag management


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8058 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-11-19 20:59:21 +00:00
apfelmaennchen
a3eebfdcba YMarks: 
- show active/running crawls
- execute crawls (works currently only if API entry is available)
- various smaller fixes

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8056 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-11-17 23:11:27 +00:00
apfelmaennchen
4f95f72124 YMarks: 
- working direct importer for YaCy Crawl Starts
- working direct import for old bookmarks.db

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8052 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-11-16 23:10:53 +00:00
apfelmaennchen
a8dfe787ed - updated to jquery flexigrid 1.1 
- YMarks.html automatically  recognizes if a bookmark is a crawl start


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8040 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-11-15 21:45:17 +00:00
apfelmaennchen
abba31f02e - bugfix for correctly sorting ymarks 
- some tuning for the autotagger (still not perfect)
- /api/ymarks/get_metadata.xml now provides info for crawlstarts
- removed unused code

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8036 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-11-14 22:00:44 +00:00
apfelmaennchen
5f7dbe1c42 - some refactoring (ymarks) 
- improvement for autotagger (is now able to create/detect  multi word tags e.g. 'open source')



git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8031 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-11-13 23:19:47 +00:00
orbiter
0d858d48ec replaced String with StringBuilder in suggestion process 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8020 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-11-09 14:42:55 +00:00
orbiter
d2ea250d99 refactoring: 
- moved many classes from de.anomic to net.yacy
- made more sub-packages for search classes

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7973 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-09-25 16:59:06 +00:00
sixcooler
59b767eebd stop loading via http at defined maximum of bytes - even size is unknown before loading 
using max-file-size of type int for parsing documents
(since content is used as byte-arrays, 'integer' should be maximum)

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7855 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-08-01 23:28:23 +00:00
orbiter
7db208c992 performance hacks: more pre-allocated StringBuilder 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7790 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-06-21 23:10:50 +00:00
orbiter
115abc8917 - more attributes for search progress bar 
- moved cache strategy to cora package

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7778 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-06-13 21:44:03 +00:00
orbiter
5b579e21a3 code cleanup 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7713 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-05-13 06:21:40 +00:00
apfelmaennchen
8b8db2aaba YMarks: some small changes/fixes 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7695 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-05-03 21:21:06 +00:00
apfelmaennchen
441035f1f4 YMarks: some improvements to flexigrid quick search on YMarks.html 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7694 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-05-02 20:11:58 +00:00
orbiter
6fa439c82b - refactoring of robots 
- added option to crawler to send error-URLs to solr
- changed solr scheme slightly (no multi-value fields where no multi values are)

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7693 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-05-02 14:05:51 +00:00
apfelmaennchen
e7c2ea193b YMark: 
- general improvements on importers, especially on auto tagging
- added get_tags (needed for tag clouds etc.)
- improved flexigrid support
- added YMarks.html (not fully working) that will eventually replace Bookmarks.html

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7691 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-05-01 21:42:48 +00:00
apfelmaennchen
b2281f0b7d YMark: intermediate work towards flexigrid support 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7670 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-04-20 22:33:01 +00:00
apfelmaennchen
60412d2bb3 YMark: 
- more refactoring >> YMarkEntry
- integration of SurrogateReader as bookmark importer
- various small bug fixes e.g. get_xbel.xml

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7668 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-04-18 21:42:14 +00:00
apfelmaennchen
a2e86daae9 YMark: more bug fixes 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7662 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-04-16 22:09:50 +00:00
apfelmaennchen
62855f9567 YMark: code clean up and some small fixes 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7661 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-04-16 21:19:42 +00:00
apfelmaennchen
667e912b19 YMark: 
- some improvements to firefox json bookmark importer
- test import with: /api/ymarks/test_import.html
- view ymarks with: /api/ymarks/test_treeview.html


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7660 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-04-16 09:09:33 +00:00
apfelmaennchen
a0e4960a4d YMark: 
- first attempt for a firefox json bookmark importer
- added JSON library json-simple-1.1.jar

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7658 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-04-15 20:58:58 +00:00