Commit Graph

6602 Commits

Author SHA1 Message Date
orbiter
474bb4de82 ups
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6730 6c8d7289-2bf4-0310-a012-ef5d649a1542
2010-03-07 23:32:18 +00:00
orbiter
8c88abf685 added follow-me link for twitter in status hints
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6729 6c8d7289-2bf4-0310-a012-ef5d649a1542
2010-03-07 23:29:29 +00:00
orbiter
58d75a6bde allow more results for a single query at the same time if the client is not authorized. This is necessary for the search widget where the default number of results is now set to 20 instead of 10 to cause that a scroll bar is shown which is necessary to get a trigger for new searches for more results.
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6728 6c8d7289-2bf4-0310-a012-ef5d649a1542
2010-03-07 22:49:20 +00:00
orbiter
57e1eae95e longer time-out for url fetching .. may help to show all that links that the statistic say for a search result
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6727 6c8d7289-2bf4-0310-a012-ef5d649a1542
2010-03-07 22:23:08 +00:00
orbiter
9e639603e3 after frequent occurrences of 100% CPU usages and permanent blockings I try to disable a function in a method that may cause the problem when calling an external library (apache http client 3.x). The thread dump that shows the problem is attached here.
at java.lang.StringCoding.encode(StringCoding.java:266)
	at java.lang.String.getBytes(String.java:946)
	at org.apache.commons.httpclient.util.EncodingUtil.getAsciiBytes(EncodingUtil.java:237)
	at org.apache.commons.httpclient.methods.multipart.Part.sendDispositionHeader(Part.java:220)
	at org.apache.commons.httpclient.methods.multipart.Part.send(Part.java:308)
	at org.apache.commons.httpclient.methods.multipart.Part.sendParts(Part.java:385)
	at org.apache.commons.httpclient.methods.multipart.MultipartRequestEntity.writeRequest(MultipartRequestEntity.java:164)
	at de.anomic.http.client.Client.zipRequest(Client.java:364)
	at de.anomic.http.client.Client.POST(Client.java:339)
	at de.anomic.yacy.yacyClient.wput(yacyClient.java:285)
	at de.anomic.yacy.yacyClient.transferURL(yacyClient.java:1053)
	at de.anomic.yacy.yacyClient.transferIndex(yacyClient.java:942)
	at de.anomic.yacy.dht.Transmission$Chunk.transmit(Transmission.java:200)
	at de.anomic.yacy.dht.Dispatcher.storeDocumentIndex(Dispatcher.java:397)
	at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:597)
	at net.yacy.kelondro.workflow.InstantBlockingThread.job(InstantBlockingThread.java:103)
	at net.yacy.kelondro.workflow.AbstractBlockingThread.run(AbstractBlockingThread.java:66)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
	at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
	at java.util.concurrent.FutureTask.run(FutureTask.java:138)
	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
	at java.lang.Thread.run(Thread.java:637)


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6726 6c8d7289-2bf4-0310-a012-ef5d649a1542
2010-03-07 21:19:23 +00:00
orbiter
4144927d94 show less errors
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6725 6c8d7289-2bf4-0310-a012-ef5d649a1542
2010-03-07 21:02:08 +00:00
mikeworks
736df39c9c Updated German translation de.lng: mainly ViewFile.html additions and removed (De)Select All from Table_API_p.html section
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6724 6c8d7289-2bf4-0310-a012-ef5d649a1542
2010-03-07 16:31:49 +00:00
orbiter
b88f5fbb4b slightly changed crawling policy
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6723 6c8d7289-2bf4-0310-a012-ef5d649a1542
2010-03-07 01:46:08 +00:00
orbiter
de01fe0e6d fix for bug in url parser
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6722 6c8d7289-2bf4-0310-a012-ef5d649a1542
2010-03-07 01:33:18 +00:00
orbiter
7684a575c4 fix for deletion of error database each time when YaCy starts up
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6721 6c8d7289-2bf4-0310-a012-ef5d649a1542
2010-03-07 00:33:39 +00:00
orbiter
f561e340c6 show more results of single domains when not authorized fully (up to 100)
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6720 6c8d7289-2bf4-0310-a012-ef5d649a1542
2010-03-07 00:12:58 +00:00
orbiter
c4bdb1e7f2 added one more option in ViewFile to show an iframe like for the orginal web page content but using the cache than the direct link to the content in the web. Upgraded the very old and previously not any more used CacheResource_p servlet to a new and working version.
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6719 6c8d7289-2bf4-0310-a012-ef5d649a1542
2010-03-06 23:41:51 +00:00
orbiter
c09a995930 better logging of double occurrences of urls in the crawler
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6718 6c8d7289-2bf4-0310-a012-ef5d649a1542
2010-03-06 20:31:30 +00:00
orbiter
1bbe14d23f SVN 6716 unfortunately contained parts of the unfinished SMB integration. To fix compile errors the remaining parts of the SMB implementation stub is added with this commit.
This adds the jcifs smb library.

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6717 6c8d7289-2bf4-0310-a012-ef5d649a1542
2010-03-05 21:46:22 +00:00
orbiter
884b262130 - added a new Wiki Namespace Navigator
- some redesign of Navigator data structures

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6716 6c8d7289-2bf4-0310-a012-ef5d649a1542
2010-03-05 21:25:49 +00:00
orbiter
b0c6d0108b fix for select-all toggle in tables servlet
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6715 6c8d7289-2bf4-0310-a012-ef5d649a1542
2010-03-05 16:15:59 +00:00
orbiter
617dfbbd06 allo 'authorization by encoded password' also if requesting client is not from localhost but from the same host as yacy is running on.
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6714 6c8d7289-2bf4-0310-a012-ef5d649a1542
2010-03-05 16:03:55 +00:00
orbiter
270fb38674 - fixed some bugs in Table viewer
- added 'select all' feature in Tables_p
- enhanced ViewFile.html: has now an input field to load arbitrary resources from the web and analyze them (!!!)
- included the ViewFile servlet into the Index Administration menu
- show in ViewFile if ressource is in url-db and/or in Web cache
- bugfixes to BEncodedHeap and Tables management

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6713 6c8d7289-2bf4-0310-a012-ef5d649a1542
2010-03-05 15:41:15 +00:00
orbiter
38d7a28cd2 fix in viewfile needed when ViewFile is called only with 'url' parameter
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6712 6c8d7289-2bf4-0310-a012-ef5d649a1542
2010-03-05 12:24:15 +00:00
orbiter
599c3766c4 added authentication to automated API call
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6711 6c8d7289-2bf4-0310-a012-ef5d649a1542
2010-03-04 14:10:03 +00:00
orbiter
727dd9b193 - fixed a bug in robots.txt parser
- moved storage of robots.txt entries to WorkTables, so it is now possible to browse the robots entries with the table browser

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6710 6c8d7289-2bf4-0310-a012-ef5d649a1542
2010-03-04 11:58:07 +00:00
orbiter
54af9e6b49 - added parsing of robots meta-tag in html headers to detect a noindexing request
- added evaluation and indexing prevention in case that a noindexing is given in a html file

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6709 6c8d7289-2bf4-0310-a012-ef5d649a1542
2010-03-03 23:32:56 +00:00
lotus
f336ed568d fir for parameters on reload
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6708 6c8d7289-2bf4-0310-a012-ef5d649a1542
2010-03-03 21:14:12 +00:00
lotus
ddc21e7b73 adapted banner colors to current standard template
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6707 6c8d7289-2bf4-0310-a012-ef5d649a1542
2010-03-03 21:10:02 +00:00
mikeworks
e76f1e6cc0 Update property name bookmarks to display_bookmarks (changed in SVN 5554) to fix the broken Bookmark RSS feed
- Fixes RSS part of http://forum.yacy-websuche.de/viewtopic.php?f=6&t=1824

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6706 6c8d7289-2bf4-0310-a012-ef5d649a1542
2010-02-28 11:22:01 +00:00
mikeworks
c52dec0c06 Updated German translation de.lng: Translated new entries in Network.html and Status_p.inc
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6705 6c8d7289-2bf4-0310-a012-ef5d649a1542
2010-02-27 03:44:25 +00:00
mikeworks
cc074c1a36 Renamed, removed and added license information for each jar archive in external lib folder
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6704 6c8d7289-2bf4-0310-a012-ef5d649a1542
2010-02-27 03:04:11 +00:00
lotus
8a19be24de tell non-windows users about the tray-icon
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6703 6c8d7289-2bf4-0310-a012-ef5d649a1542
2010-02-26 19:44:23 +00:00
orbiter
f5ec7ad077 replaced four old libraries with latest version
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6702 6c8d7289-2bf4-0310-a012-ef5d649a1542
2010-02-26 14:14:50 +00:00
orbiter
6b89f681c5 added bing, dbpedia and wolfram alpha to the compare-search options
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6701 6c8d7289-2bf4-0310-a012-ef5d649a1542
2010-02-26 10:34:36 +00:00
mikeworks
475ffabfa1 Added License (Apache 1.1) information for Jakarta ORO library 2.0.7 (2.0.8 available)
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6700 6c8d7289-2bf4-0310-a012-ef5d649a1542
2010-02-26 04:26:04 +00:00
orbiter
46c4f8b68a better look-ahead into the crawl queue: show more on crawl monitor
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6699 6c8d7289-2bf4-0310-a012-ef5d649a1542
2010-02-24 23:11:58 +00:00
orbiter
4b6efe3b48 more ergonomic default values for crawl start
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6698 6c8d7289-2bf4-0310-a012-ef5d649a1542
2010-02-24 22:33:51 +00:00
lotus
7b546415dc added svn6695 for windows
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6697 6c8d7289-2bf4-0310-a012-ef5d649a1542
2010-02-24 14:58:53 +00:00
orbiter
f175f9a2d3 changed way how number of search requests are counted:
so far only search requests at the remote search interface had been counted.
This was done to protect the privacy of searchers, because counting was not done and published at the own search interface.
This caused that no search requests of robinson peers had been counted, becuase they cannot be counted at remote peer.
This change introduces a distinction of locally done search requests at the local search interface from search requests that are on the local interface but had been submitted from a remote IP without authentication.
Now 3 counters are maintained:
- partial count of remote searches
- total count of local searches on robinson peers from non-authenticated clients
- total count of local searches on robinson peers from localhost or authenticated clients
In the global statistic of search requests now the first two counters of the three cases are added
Because we habe a large number of robinson peers with a large number of remote non-authenticated requests the statistic should show at least three times of the number of search requests.


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6696 6c8d7289-2bf4-0310-a012-ef5d649a1542
2010-02-24 13:53:55 +00:00
orbiter
84222e3b4f fix for auto-updater: delete old libraries before copy of new one
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6695 6c8d7289-2bf4-0310-a012-ef5d649a1542
2010-02-24 13:46:50 +00:00
sixcooler
cd6de83905 next try for for http://forum.yacy-websuche.de/viewtopic.php?f=5&t=2703
(reverted 6692)

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6694 6c8d7289-2bf4-0310-a012-ef5d649a1542
2010-02-23 15:59:58 +00:00
sixcooler
bfe4693e9a fix for http://forum.yacy-websuche.de/viewtopic.php?f=5&t=2703
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6693 6c8d7289-2bf4-0310-a012-ef5d649a1542
2010-02-23 13:46:56 +00:00
orbiter
6fde481ef4 missing for last commit
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6692 6c8d7289-2bf4-0310-a012-ef5d649a1542
2010-02-22 20:21:39 +00:00
orbiter
b68deb407a - moved test data from /bin to /test/words
- refactoring of stopYACY.sh by introduction of /bin/apicall which is able to call any api file with attached authorization

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6691 6c8d7289-2bf4-0310-a012-ef5d649a1542
2010-02-22 20:14:16 +00:00
orbiter
cd7f0bf75f wait a little bit longer for a init.d restart between stop and start
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6690 6c8d7289-2bf4-0310-a012-ef5d649a1542
2010-02-22 15:42:48 +00:00
mikeworks
50169759ca Replaced old pdfbox and fontbox LICENSE files with new ones (still Apache 2.0)
Testing delete and adding files

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6689 6c8d7289-2bf4-0310-a012-ef5d649a1542
2010-02-22 07:09:57 +00:00
orbiter
1e2c011c98 updated the jsch lib from 0.1.21 to 0.1.42
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6688 6c8d7289-2bf4-0310-a012-ef5d649a1542
2010-02-21 23:43:50 +00:00
orbiter
c2b505ae87 updated bouncy castle libraries
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6687 6c8d7289-2bf4-0310-a012-ef5d649a1542
2010-02-21 23:31:40 +00:00
orbiter
681f4d185f replaced microsoft office document parser POI 3.5 with latest version 3.6
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6686 6c8d7289-2bf4-0310-a012-ef5d649a1542
2010-02-21 23:18:52 +00:00
orbiter
e9cdddcd0f updated parser libraries fontbox and pdfbox with latest version of jar files
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6685 6c8d7289-2bf4-0310-a012-ef5d649a1542
2010-02-21 23:05:38 +00:00
orbiter
93b7ddc27d fix for http://forum.yacy-websuche.de/viewtopic.php?p=19376#p19376
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6684 6c8d7289-2bf4-0310-a012-ef5d649a1542
2010-02-21 22:49:35 +00:00
lotus
3df1060015 jre update für installer
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6683 6c8d7289-2bf4-0310-a012-ef5d649a1542
2010-02-20 08:59:17 +00:00
mikeworks
7d569ce735 Updated German translation de.lng: Translated new buttons in Table_API_p.html
Updated French translation fr.lng: cleaned up syntax and added a few lines
Updated Italian translation it.lng: cleaned up syntax
Updated Slovakian translation sk.lng: cleaned up syntax

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6682 6c8d7289-2bf4-0310-a012-ef5d649a1542
2010-02-19 04:24:58 +00:00
orbiter
23bf81d225 new buttons to select/deselect all entries
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6681 6c8d7289-2bf4-0310-a012-ef5d649a1542
2010-02-18 23:31:23 +00:00