Commit Graph

7431 Commits

Author SHA1 Message Date
orbiter
0960261769 fix for svn 7713
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7715 6c8d7289-2bf4-0310-a012-ef5d649a1542
2011-05-13 15:20:57 +00:00
apfelmaennchen
7e368000c8 transparent progress bar
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7714 6c8d7289-2bf4-0310-a012-ef5d649a1542
2011-05-13 13:40:23 +00:00
orbiter
5b579e21a3 code cleanup
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7713 6c8d7289-2bf4-0310-a012-ef5d649a1542
2011-05-13 06:21:40 +00:00
orbiter
fcd4b03892 show progress of search after display of results is finished
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7712 6c8d7289-2bf4-0310-a012-ef5d649a1542
2011-05-13 06:20:00 +00:00
lotus
8b63d7637d revert 7710,
configure java in firewall for full access. this is the usual config if the user manually accepts java for the firewall.
otherwise if a port is specified, windows will not ask on any port change. this would break yacy and other java applications if they run not on the specified port. this would not be an expected behaviour for the user.

firewall config may fail for win64 systems (system32 is specified in path)

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7711 6c8d7289-2bf4-0310-a012-ef5d649a1542
2011-05-09 20:06:05 +00:00
pca
440e3ba887 Windows Installer:
- remove firewall-handling for WinXP (can only open for JRE not for 
  special port)
- Vista/Win 7: open port 1900 for communication with router (uPnP)

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7710 6c8d7289-2bf4-0310-a012-ef5d649a1542
2011-05-09 06:16:35 +00:00
orbiter
039126cfaf better handling of on/off switched solr indexing
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7709 6c8d7289-2bf4-0310-a012-ef5d649a1542
2011-05-08 22:47:20 +00:00
orbiter
dc54915df4 fix for very bad compare
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7708 6c8d7289-2bf4-0310-a012-ef5d649a1542
2011-05-08 08:45:58 +00:00
lotus
f123dbec79 fix in heuristics config
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7707 6c8d7289-2bf4-0310-a012-ef5d649a1542
2011-05-07 18:52:20 +00:00
orbiter
897b4e8b9c another hack to prevent black images
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7706 6c8d7289-2bf4-0310-a012-ef5d649a1542
2011-05-07 07:45:02 +00:00
orbiter
9248a4eef4 reduce teh effect of 'Bildersuche findet generierte HTML-Seiten als Bilder'
see http://bugs.yacy.net/view.php?id=9

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7705 6c8d7289-2bf4-0310-a012-ef5d649a1542
2011-05-07 07:37:46 +00:00
orbiter
0621a15f89 fix for wrong search result counter: added a counter for all filtered out entities
see also http://bugs.yacy.net/view.php?id=5

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7704 6c8d7289-2bf4-0310-a012-ef5d649a1542
2011-05-06 23:04:27 +00:00
apfelmaennchen
61c9a791c4 YMarks: sidebar with tabs for tags and folders
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7703 6c8d7289-2bf4-0310-a012-ef5d649a1542
2011-05-06 21:36:35 +00:00
orbiter
9c33b2fb58 fix for String Matcher in case that no snippet is returned (NPE)
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7702 6c8d7289-2bf4-0310-a012-ef5d649a1542
2011-05-05 23:11:03 +00:00
orbiter
76f2817e00 a fix for the snippet computation and hopefully better snippets
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7701 6c8d7289-2bf4-0310-a012-ef5d649a1542
2011-05-05 23:05:38 +00:00
orbiter
deda54d684 - relaxed matching of string-search (this is now case-insensitive)
- added transport of string-search pattern to remote search protocol
- fixed a problem parsing snippets with a '-' inside

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7700 6c8d7289-2bf4-0310-a012-ef5d649a1542
2011-05-05 22:37:06 +00:00
lotus
8fd4e8ea98 proper jre version (without -s in filename)
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7699 6c8d7289-2bf4-0310-a012-ef5d649a1542
2011-05-05 20:03:27 +00:00
orbiter
15e3a57b4e removed unused functions in condenser
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7698 6c8d7289-2bf4-0310-a012-ef5d649a1542
2011-05-05 09:23:10 +00:00
orbiter
6e42d4de88 - added full-String search function: find things that match exactly what is quoted in the query
- re-structuring authentification methods to fix a problem with API steering

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7697 6c8d7289-2bf4-0310-a012-ef5d649a1542
2011-05-05 00:25:14 +00:00
orbiter
8e10b82280 small fix for solr export
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7696 6c8d7289-2bf4-0310-a012-ef5d649a1542
2011-05-03 22:21:45 +00:00
apfelmaennchen
8b8db2aaba YMarks: some small changes/fixes
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7695 6c8d7289-2bf4-0310-a012-ef5d649a1542
2011-05-03 21:21:06 +00:00
apfelmaennchen
441035f1f4 YMarks: some improvements to flexigrid quick search on YMarks.html
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7694 6c8d7289-2bf4-0310-a012-ef5d649a1542
2011-05-02 20:11:58 +00:00
orbiter
6fa439c82b - refactoring of robots
- added option to crawler to send error-URLs to solr
- changed solr scheme slightly (no multi-value fields where no multi values are)

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7693 6c8d7289-2bf4-0310-a012-ef5d649a1542
2011-05-02 14:05:51 +00:00
sixcooler
1ea0bc775c @apfelmaenchen:
is this the expected, but forgotten change?
Please correct if I'm wrong
(this let me build Yacy again)

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7692 6c8d7289-2bf4-0310-a012-ef5d649a1542
2011-05-02 10:46:05 +00:00
apfelmaennchen
e7c2ea193b YMark:
- general improvements on importers, especially on auto tagging
- added get_tags (needed for tag clouds etc.)
- improved flexigrid support
- added YMarks.html (not fully working) that will eventually replace Bookmarks.html

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7691 6c8d7289-2bf4-0310-a012-ef5d649a1542
2011-05-01 21:42:48 +00:00
orbiter
e3d19d0a90 fix in Document inboundlinks/outboundlinks sorting
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7690 6c8d7289-2bf4-0310-a012-ef5d649a1542
2011-05-01 15:49:04 +00:00
pca
5e2d38ef19 Windows Installer:
- fix for firewall Vista/Win7
- update to JRE 1.6 u25
- TODO: fix for firewall WinXP and setting for uPnP (Port 1900)

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7689 6c8d7289-2bf4-0310-a012-ef5d649a1542
2011-04-30 19:32:07 +00:00
orbiter
4e8fa03514 added more attributes to html evaluation
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7688 6c8d7289-2bf4-0310-a012-ef5d649a1542
2011-04-29 15:36:44 +00:00
orbiter
3b578a28ef some patches to prevent that empty or bad IP information is broadcasted
- on client-side: fix bad IP reports from remote Peers by replacing their reported IP with their server IP if the reported IP is bad, broken or disallowed
- on server-side: the same during a peer ping (here the ping'ed server acts also as client during the back-ping) and also when receiving a message or a search where the client sends also its seed. Here the IP is replaced by the client IP if the reported IP is broken or bad

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7687 6c8d7289-2bf4-0310-a012-ef5d649a1542
2011-04-29 10:58:12 +00:00
orbiter
361841df16 another patch according to http://bugs.yacy.net/view.php?id=26#c36
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7686 6c8d7289-2bf4-0310-a012-ef5d649a1542
2011-04-29 02:26:50 +00:00
orbiter
37fede9d30 better logic for proper seed ip recognition and better error messages
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7685 6c8d7289-2bf4-0310-a012-ef5d649a1542
2011-04-29 02:19:13 +00:00
orbiter
8b95a26866 better magic
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7684 6c8d7289-2bf4-0310-a012-ef5d649a1542
2011-04-29 02:00:37 +00:00
orbiter
2700a58e5a added a magic to the peer ping that will be used in case that the contacting peer requests that it's reported IP shall be used for a back-ping. The back-ping now also returns the same magic which will make it possible that the requested peer can verify that the back-pinged peer is actually the same peer.
This is also a protection against the foced-fake of a external IP: if such an IP was faked, then the next ping from the affected peer to another peer looks like a staticIP report. Such a bad staticIP-by-faked-response can now be discovered and fixed by the peer that gets the second ping after the first ping contained a faked response.

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7683 6c8d7289-2bf4-0310-a012-ef5d649a1542
2011-04-29 01:52:20 +00:00
orbiter
8879cc1db2 removed System.out.println
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7682 6c8d7289-2bf4-0310-a012-ef5d649a1542
2011-04-28 14:08:02 +00:00
orbiter
c493f101c0 added one more script file to release build script
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7681 6c8d7289-2bf4-0310-a012-ef5d649a1542
2011-04-28 13:19:24 +00:00
orbiter
528da7c9ea removed unused class and added license header for new class
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7680 6c8d7289-2bf4-0310-a012-ef5d649a1542
2011-04-28 13:14:30 +00:00
orbiter
f6077b3cc0 added more attributes for html parser and enhanced data structures
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7679 6c8d7289-2bf4-0310-a012-ef5d649a1542
2011-04-28 13:09:01 +00:00
f1ori
0b02083e97 * function for simple crawl of one url
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7678 6c8d7289-2bf4-0310-a012-ef5d649a1542
2011-04-28 13:04:33 +00:00
f1ori
d671de8c17 add ranking weight to json-search-results
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7677 6c8d7289-2bf4-0310-a012-ef5d649a1542
2011-04-28 11:18:14 +00:00
sixcooler
4eb9c1e7c3 not setting userAgent from Constructor as default for following calls
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7676 6c8d7289-2bf4-0310-a012-ef5d649a1542
2011-04-26 17:39:16 +00:00
orbiter
d8e934c085 better abstraction of http client identification
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7675 6c8d7289-2bf4-0310-a012-ef5d649a1542
2011-04-26 13:35:29 +00:00
sixcooler
a3e707283d not using HTTPConnector anymore
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7674 6c8d7289-2bf4-0310-a012-ef5d649a1542
2011-04-26 11:46:31 +00:00
orbiter
9f1f47ec67 added some comments to explain the isLocal patch
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7673 6c8d7289-2bf4-0310-a012-ef5d649a1542
2011-04-21 21:59:56 +00:00
orbiter
b77b8cac0c - enhanced html parser: recognized much more details in the content
- added more properties to solr index
- refactoring
- more constants in switchboard
- fix for some NPEs
- recognition of more images
- removed synchronization in HandleMap (obviously not necessary?)
- added a nolocal configuration to remove excessive dns lookup (works only on allip - default off). Indexes produced with this setting are all flagged with 'local' and are (on purpose) not usable for freeworld because they will be rejected as beeing local.



git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7672 6c8d7289-2bf4-0310-a012-ef5d649a1542
2011-04-21 13:58:49 +00:00
low012
bc84d2bc9d *) fixed typo in stop script
*) added <u> </u> tags for underlined text in Wiki Code
*) minor code changes

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7671 6c8d7289-2bf4-0310-a012-ef5d649a1542
2011-04-20 22:54:29 +00:00
apfelmaennchen
b2281f0b7d YMark: intermediate work towards flexigrid support
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7670 6c8d7289-2bf4-0310-a012-ef5d649a1542
2011-04-20 22:33:01 +00:00
low012
06d50fd801 *) fixed stupid bug (introduced in r7663 by myself) which caused wrong parsing of Wiki pages
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7669 6c8d7289-2bf4-0310-a012-ef5d649a1542
2011-04-20 17:27:59 +00:00
apfelmaennchen
60412d2bb3 YMark:
- more refactoring >> YMarkEntry
- integration of SurrogateReader as bookmark importer
- various small bug fixes e.g. get_xbel.xml

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7668 6c8d7289-2bf4-0310-a012-ef5d649a1542
2011-04-18 21:42:14 +00:00
low012
7c149e0f9d *) ./stopYACY:sh -f kills YaCy in case regular shutdown does not work
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7667 6c8d7289-2bf4-0310-a012-ef5d649a1542
2011-04-18 19:09:54 +00:00
orbiter
3d5104d357 - fixed a bug in crawl start with file name (npe in new url)
- added deletion of solr index in IndexControlRWIs
- added asynchronous adding of large url lists (happens when crawls are startet with file)
- fixed npe in Image display
- replaced language warning with fine logging
- added a domain name cache in Domains that helps to speed up the isLocal property (less DNS lookups)
- added a new storage class for this new cache: KeyList. The domain key list is stored in DATA/WORK/globalhosts.list
- added concurrent solr updates and chunked transfers (50 documents until a commit is done) for high-speed feeding (> 40000 ppm)
- fixed a bug in content scraper that chopped off large parts of crawl lists (using crawl start from file)

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7666 6c8d7289-2bf4-0310-a012-ef5d649a1542
2011-04-18 16:11:16 +00:00