Marown/yacy_search_server
1
0
Fork 0
mirror of https://github.com/yacy/yacy_search_server.git synced 2024-09-19 00:01:41 +02:00
Code Issues Actions Packages Projects Releases Wiki Activity
8,289 Commits 1 Branch 20 Tags 836 MiB
b2175ea4ef
Commit Graph

235 Commits

Author SHA1 Message Date
reger
b2175ea4ef Add possibility to set custom Solr field names for the YaCy default Solr attributes. 
- Changing the format of YaCy's solr.key.list while maintainig backward compatibility
  Federated index config screens adjusted accordingly
- modified the Solr update request to use a 3 min Solr autocommit intervall
 2012-05-15 22:34:02 +02:00
Michael Peter Christen
c00efc2717 made the solr connection more generic 2012-05-09 16:46:45 +02:00
Michael Peter Christen
453010bd68 - solved problems with backpath normalization 
- redesigned in/outbound link handover
- removed iframe links from inbound/outbound in solr scheme
 2012-04-27 16:48:51 +02:00
Michael Peter Christen
0e13022147 - enhanced solr field documentation 
- added xml api button to IndexFederated_p - the solr schema.xml file
can be generated by YaCy
 2012-04-26 15:25:07 +02:00
Michael Peter Christen
e377092198 fix to xml output format 2012-04-13 09:02:18 +02:00
Michael Christen
41be98dc9d extended webstructure api to show together with incoming links also 
outgoing links
 2012-04-13 11:53:34 +02:00
Michael Christen
8f89c8ef07 added information about inbound, outbound and citation links into 
yacydoc api servlet
 2012-03-31 07:38:49 +02:00
Michael Christen
71649a1296 added an api to retrieve the new citation.index with the 
webstructure.xml api. This api will respond with details about a single
URL if requested with 'webstructure.xml?about=[url|urlhash|host]'.
 2012-03-29 17:22:31 +02:00
Michael Peter Christen
9ad1d8dde2 complete redesign of crawl queue monitoring: do not look at a 
ready-prepared crawl list but at the stacks of the domains that are
stored for balanced crawling. This affects also the balancer since that
does not need to prepare the pre-selected crawl list for monitoring. As
a effect:
- it is no more possible to see the correct order of next to-be-crawled
links, since that depends on the actual state of the balancer stack the
next time another url is requested for loading
- the balancer works better since the next url can be selected according
to the current situation and not according to a pre-selected order.
 2012-02-02 21:33:42 +01:00
Michael Peter Christen
e2f8f263e8 changed storage of search words: keep order 2012-02-01 18:13:31 +01:00
Michael Peter Christen
c166eb68b6 fixes in solr schema file 2012-02-01 00:22:43 +01:00
Lotus
335a776351 xss hardening on Status.html 2012-01-28 13:25:12 +01:00
Michael Peter Christen
ef5192f8c9 using the generic document parser for crawl starts instead of the html 
parser. This makes it possible that every type of document can be a
crawl start point, not only text documents or html documents. Testet
this with a pdf document.
 2012-01-23 17:27:29 +01:00
Michael Peter Christen
ce620be783 for for crawl start with smb url 2012-01-19 23:07:15 +01:00
Michael Peter Christen
7053f8ab46 added automatic generation of a solr schema.xml file 2012-01-13 11:26:22 +01:00
Michael Peter Christen
2ee8cbeb2c Merge branch 'master' of ssh://git@gitorious.org/yacy/rc1.git 
Conflicts:
	source/net/yacy/search/Switchboard.java
 2012-01-05 18:37:46 +01:00
Michael Peter Christen
992dbdf4bb added noload statistic to servlets 2012-01-05 18:33:05 +01:00
Roland 'Quix0r' Haeder
fa08ed5ae5 Fixed a lot CHMOD rights (no need for execute flag on *.java/*.html) and introduced local/remote crawl size ratio based check 2011-12-29 00:33:16 +01:00
Michael Christen
9e5894c784 Removed handling of components objects for URIMetadataRows. 
This is a preparation to replace this rows with nodes from the node
store.
 2011-12-17 01:27:08 +01:00
Michael Christen
c04bfaa51b refactoring 2011-12-16 23:59:29 +01:00
Michael Christen
e9dc99fe15 added rules to set specific RWIs as private RWIs which are not 
transmitted to remote peers. This will be used for private index copies
and phonetic indexes.
 2011-12-14 22:15:51 +01:00
Michael Christen
204c29f010 small bugfixes for search result display and cache display 2011-12-10 01:35:38 +01:00
apfelmaennchen
ff19fcdb28 bugfix for YMarks XBEL import and export; thanks to Dominic 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8138 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-12-06 23:17:57 +00:00
orbiter
11729061f2 added an option in the bookmark import process to put everything into the crawler 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8134 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-12-03 00:27:01 +00:00
apfelmaennchen
8f30d288e9 small change to mouse over text for crawl starts within bookmarks 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8131 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-12-02 09:43:34 +00:00
apfelmaennchen
29e97f94f2 small optical enhancements to ymarks treeview 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8106 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-11-25 18:08:42 +00:00
apfelmaennchen
77a080ced9 smaller fixes for YMarks 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8105 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-11-25 17:33:03 +00:00
orbiter
e22f8497c9 - tested the ARC methods 
- removed strict authentication (if password is empty; this was buggy and not useful; can be switched on if necessary globally and not for each interface method)
- increased speed of CrawlResults page (no dns lookup any more)
- increased speed of favicon display (removed dns lookup)

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8104 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-11-25 14:09:25 +00:00
orbiter
5a55397f99 some last-minute performance hacks 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8101 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-11-25 11:23:52 +00:00
apfelmaennchen
dd1482aaf5 further update to YMarks 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8100 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-11-25 00:32:18 +00:00
apfelmaennchen
564374d1fe - included YMarks in addition to old bookmarks in yacysearchitem.html; don't get confused by the old bookmark dialog, the ymark is automatically added silently beforehand. 
- reworked bookmark creation on crawlstart
- many smaller adjustments to ymarks


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8072 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-11-22 23:50:49 +00:00
orbiter
05f34a3fa7 added a full, complete, database insert, update and delete API for the tables. 
Please see this example:

list all database tables:
http://localhost:8090/api/table_p.xml

now create a new table and insert some values into 'mytable'
http://localhost:8090/api/table_p.xml?table=mytable&pk=&commitrow=&col_termin=Release%20Machen&col_datum=24.11.2011&col_status=ongoing

list the table content:
http://localhost:8090/api/table_p.xml?table=mytable&pk=

update the table and change a single value inside. You must refer to the row using a primary key 'pk'
http://localhost:8090/api/table_p.xml?table=mytable&pk=000000000001&commitrow=&col_datum=29.11.2011

you can also select rows using a search operator
http://localhost:8090/api/table_p.xml?table=mytable&pk=&count=10&search=

now lets delete the row:
http://localhost:8090/api/table_p.xml?table=mytable&pk=&deleterows=pk_000000000001

and we can also delete the complete table:
http://localhost:8090/api/table_p.xml?table=mytable&deletetable=

You can use this to administrate the robots, bookmarks and API steering using an outside application!

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8071 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-11-22 12:31:07 +00:00
orbiter
c461c1eebf fixed xml output for table retrieval 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8065 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-11-22 10:18:13 +00:00
orbiter
c93f10417a add a bookmark automatically each time a new crawl is started 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8063 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-11-22 00:03:20 +00:00
apfelmaennchen
6287c2b4a9 YMarks: 
- introduced tag manager - a quite powerful tool (still not 100% stable, so be careful)

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8060 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-11-20 22:42:15 +00:00
apfelmaennchen
5581be12fb YMarks: 
- added backend and api for tag management


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8058 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-11-19 20:59:21 +00:00
apfelmaennchen
a3eebfdcba YMarks: 
- show active/running crawls
- execute crawls (works currently only if API entry is available)
- various smaller fixes

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8056 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-11-17 23:11:27 +00:00
apfelmaennchen
4f95f72124 YMarks: 
- working direct importer for YaCy Crawl Starts
- working direct import for old bookmarks.db

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8052 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-11-16 23:10:53 +00:00
orbiter
017a01714d - enhanced logging in robots.txt parser for remote debugging 
- robots.txt is now more robust against database operations

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8043 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-11-16 01:03:49 +00:00
orbiter
5a7cec59f3 moved ynetSearch to get all files out of htroot/api/util/ 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8042 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-11-16 00:21:56 +00:00
apfelmaennchen
a8dfe787ed - updated to jquery flexigrid 1.1 
- YMarks.html automatically  recognizes if a bookmark is a crawl start


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8040 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-11-15 21:45:17 +00:00
cominch
cef8ebc41d getpageinfo: Checks if there is a OAI repository behind the URL. 
This check is only performed if oai parameter is set when calling e.g. getpageinfo_p.xml?actions=oai

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8039 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-11-15 12:22:19 +00:00
orbiter
eb1c7c041d write info about robots.txt evaluation into getpageinfo_p.xml 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8038 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-11-15 00:33:54 +00:00
orbiter
f8b8c82421 - refactoring of getpageinfo_p.xml (moved out of util) 
- added more logging in getpageinfo_p.xml

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8037 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-11-15 00:22:40 +00:00
apfelmaennchen
abba31f02e - bugfix for correctly sorting ymarks 
- some tuning for the autotagger (still not perfect)
- /api/ymarks/get_metadata.xml now provides info for crawlstarts
- removed unused code

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8036 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-11-14 22:00:44 +00:00
apfelmaennchen
5f7dbe1c42 - some refactoring (ymarks) 
- improvement for autotagger (is now able to create/detect  multi word tags e.g. 'open source')



git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8031 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-11-13 23:19:47 +00:00
apfelmaennchen
4d7ae76017 - update to jquery 1.7 (does not apply to all jquery code, old version is additionally kept for compatibility) 
- update to jquery-ui 1.8.16 (includes themes)
- introduced new portalsearch (as default)
- old portalsearch is still available and accessible, but will eventually be removed
- jquery and portal search is now loaded by special header templates for maintenance reasons
- update to new autocomplete, solves bug: http://bugs.yacy.net/view.php?id=29
- many improvements to YMarks GUI and API...more to come anytime soon

Sorry, this is a rather large commit, I hope it doesn't break anything essential, but I need to consolidate some of my efforts in order to move ahead. Especially the update to the portalsearch widget might not be welcomed, but the old one is simply incompatible with newer jquery and jquery-ui libraries, sorry. The code tree /yacy/ui/... is obsolete and will be removed in the future. At that point all productive portalsearches should have migrated to the new version.



git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8014 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-11-07 20:44:58 +00:00
orbiter
a7df70221e refactoring 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7987 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-10-04 09:06:24 +00:00
orbiter
d2ea250d99 refactoring: 
- moved many classes from de.anomic to net.yacy
- made more sub-packages for search classes

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7973 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-09-25 16:59:06 +00:00
orbiter
2d03dc1804 removed unnecessary warning 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7916 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2011-09-01 10:37:14 +00:00