Marown/yacy_search_server
1
0
Fork 0
mirror of https://github.com/yacy/yacy_search_server.git synced 2024-09-19 00:01:41 +02:00
Code Issues Actions Packages Projects Releases Wiki Activity
5,548 Commits 1 Branch 20 Tags 836 MiB
64a63306b8
Commit Graph

3662 Commits

Author SHA1 Message Date
orbiter
16f5c6a85e fixed merge method initialization in ReferenceContainer 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5676 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-03-07 10:45:14 +00:00
orbiter
d7a493b4f5 added experimental timeline api 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5672 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-03-06 16:01:29 +00:00
orbiter
efcd95dc37 simplification of (internal) query process / refactoring 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5671 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-03-06 15:53:20 +00:00
orbiter
f1b712c29a small corrections to image loading methods in result presentation 
especially loading of favicons in search results. This is a fix that
affects only searches in intranet/repository configurations.

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5670 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-03-06 15:39:02 +00:00
orbiter
d4b56d5819 added more asserts to BLOBHeap.flushBuffer() to fix the problem described in 
http://forum.yacy-websuche.de/viewtopic.php?f=6&t=1679&hilit=&p=13109#p13109

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5666 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-03-03 23:24:19 +00:00
f1ori
c545fcb9fa * add class to handle keys and signatures 
* fix bug in serverCharBuffer
* add build-target to sign tar.gz (run ant dist sign)


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5665 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-03-02 13:29:50 +00:00
orbiter
aa44d9bad9 more refactoring of kelondro.text / deleted de.anomic.index 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5664 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-03-02 11:04:13 +00:00
orbiter
6ffc6e3389 more refactoring of indexer and kelondro classes; 
- integrating the indexer into kelondro as package 'text'
- renaming of classes in kelondro.index

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5663 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-03-02 10:00:32 +00:00
orbiter
404bc21da9 simplification of (internal) query process / refactoring 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5662 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-03-02 08:48:27 +00:00
orbiter
76ef5f0f14 refactoring of index package: better names for the classes (to be continued) 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5661 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-03-01 23:58:14 +00:00
orbiter
2df57b1fd1 refactoring of index collection class 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5660 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-03-01 23:07:45 +00:00
lotus
39a177649b * added upnp listener for devices that do not respond to discovery but advertise themselves 
* moved package

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5659 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-02-28 14:36:23 +00:00
orbiter
d1d9fbae5c enabling the URLAnalysis to operate on multime input files, just use a wild card when calling the class from the command line 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5658 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-02-26 23:47:41 +00:00
orbiter
c728879ab8 fixes to yacyURL - more exceptions in case that urls are strange 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5657 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-02-26 22:33:47 +00:00
orbiter
7542336ae5 performance enhancement to yacyURL: omit second processing of resolveBackpath. This method is already applied during initialization of the object and was called a second time when the url was exportet. 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5656 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-02-26 21:52:32 +00:00
orbiter
7ea53fe47b added another url list transformation option: 
- check the list and kick out entries with lines that contain not valid urls
- normalize the urls
- remove doubles
- sort the list
- split the list in smaller chunks
This is all done in one process which can be called with a new -sort option

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5655 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-02-26 21:51:23 +00:00
orbiter
e521e81148 bugfix in yacyURL (for latest performance hack) 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5654 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-02-26 07:46:47 +00:00
orbiter
54625360f7 performance update 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5653 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-02-25 23:27:21 +00:00
orbiter
d884c4718a added gzip support for URLAnalysis: 
url lists can also be compressed with gzip
If such a file is handed over to URLAnalysis, the output will also be written as .gz-file

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5652 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-02-25 13:40:51 +00:00
orbiter
46632f4385 performance update to yacyURL 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5651 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-02-25 00:17:34 +00:00
orbiter
cf9b74e6e3 added another method to process url lists: extract hosts only 
This can be used like
java -Xmx2000m -cp classes de.anomic.data.URLAnalysis -host DATA/EXPORT/20090224213823.txt

changed als the call method to generate statistics, please use now
java -Xmx2000m -cp classes de.anomic.data.URLAnalysis -stat DATA/EXPORT/20090224213823.txt


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5650 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-02-24 22:51:07 +00:00
orbiter
89d8e824ed memory protection for URLAnalysis 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5649 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-02-24 22:05:09 +00:00
orbiter
0f6fa804ff performance update to URLAnalysis 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5648 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-02-24 21:35:33 +00:00
orbiter
8444357291 added new row interator in kelondro tables files that enumerates rows 
without an order by the primary key. The result is a very fast enumeration of the Eco table data structure. Other table data types are not affected.
The new enumerator is used for the url export function that can be accessed from the online interface (Index Administration -> URL References -> Export). This export should now be much faster, if all url database files are from type Eco
The new enumeration is also used at other functions in YaCy, i.e. the initialization of the crawl balancer and the initialization of YaCy News.

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5647 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-02-24 10:40:20 +00:00
orbiter
e8f5f2f612 added tool to analyse url strings 
and to generate statistics about words occurring in urls

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5646 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-02-24 10:00:35 +00:00
lotus
6117e083e5 option to customize tray label (tooltip) with tray.label 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5642 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-02-23 21:07:08 +00:00
orbiter
b8c3803bfc don't panic when canceling server sessions 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5641 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-02-23 17:15:52 +00:00
orbiter
de714783b1 - added host, path, filename to search result 
- modified yacyinteractive, shows now also date
- added size attribut to export file in xml format

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5639 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-02-23 11:39:20 +00:00
lotus
9519d84372 changed "dooble" variable to "browserintegration" to be less specific 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5636 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-02-22 17:32:17 +00:00
lotus
8429083972 adjusted tray for dooble: 
you can now set dooble=true in yacy.init to disable the menu and browser popups by default

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5633 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-02-22 15:11:44 +00:00
orbiter
ef62ec635e removed overwriting of logging config 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5629 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-02-21 00:43:38 +00:00
orbiter
c852d2d70e - reject too old seeds 
- do not store the complete seed in the reverse name cache, only the hash of the peer

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5628 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-02-21 00:39:44 +00:00
orbiter
aca973e2d9 catch more exceptions 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5627 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-02-20 23:24:49 +00:00
orbiter
9559bc23fd automatic clean-up of dead connections 
(hope that works well..)

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5626 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-02-20 22:27:02 +00:00
hermens
02dfd6183b Fix logging in serverCore 
Prevent NPEs from keeping stopped Sessions in the pool and blocking slots



git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5625 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-02-20 18:54:01 +00:00
hermens
d30456e2c8 Fix logging in serverCore 
Prevent NPE:
I 2009/02/20 15:15:56 PLASMA check for Session_77.37.19.225:38812#0: 86515 ms alive, stopping thread
I 2009/02/20 15:15:56 PLASMA Closing main socket of thread 'Session_77.37.19.225:38812#0'
E 2009/02/20 15:15:56 SERVER receive interrupted - exception 2 = Socket closed
Exception in thread "Session_77.37.19.225:38812#0" java.lang.NullPointerException
        at de.anomic.server.serverCore$Session.run(serverCore.java:623)



git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5624 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-02-20 15:12:00 +00:00
orbiter
4f9dae2571 remove reference in crawl entries 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5623 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-02-19 22:58:00 +00:00
orbiter
1ba4301920 automated interruption of dead incoming connections, if they are there for more than one minute 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5622 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-02-19 22:27:24 +00:00
orbiter
c12bb8a6d0 - refactoring of the http client 
- added a protection against memory leaks for the access tracker

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5621 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-02-19 16:24:46 +00:00
orbiter
5d3983faae the soLinger parameter was wrong. 
With soLinger=true the httpd looses connections
The effect can be seen when crawling the internal repository:
lost connections filled the client process queue until it was full
and no more connections were possible.

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5620 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-02-17 16:22:15 +00:00
orbiter
62505bb3cb more bugfixes as recommendet by findbugs 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5619 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-02-17 09:12:47 +00:00
orbiter
6b450d09ca some fixes recommended by findbugs 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5618 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-02-16 23:31:54 +00:00
orbiter
4db80065ac select more 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5617 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-02-16 21:53:37 +00:00
orbiter
94c42691d8 - reject less transmissions as transmission receiver 
- do not flag too much receiver when something goes wrong during transmission as sender

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5616 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-02-16 21:28:48 +00:00
orbiter
f887fc159f try to reduce the large number of unclosed incoming connections 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5615 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-02-16 16:26:57 +00:00
orbiter
e04a0e05c3 fix for last commit 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5614 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-02-16 16:21:12 +00:00
orbiter
a9ad863686 second part of 'doubles' fix - better handling of doubles in RAMIndex. More logging. 
still missing: deletion of double entries in collections

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5613 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-02-16 16:13:48 +00:00
orbiter
59427064fb first part of 'doubles' fix (not fully ready yet) 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5612 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-02-16 00:47:48 +00:00
orbiter
26978b2a25 - better memory protection in kelondro caches: computation of needed memory for cache grow 
- removed excessive gc calls
- step to 16 vertical DHT partitions

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5611 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-02-15 23:35:59 +00:00
lotus
e9e2fff47a better scaling on performance graph 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5610 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-02-15 17:36:13 +00:00
lotus
4aad461100 added UPnP support 
YaCy can now automatically forward ports on home routers
off by default

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5609 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-02-14 13:12:08 +00:00
orbiter
99b9788e54 fix for possible 100% CPU caused by concurrent access of HashMap 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5607 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-02-14 00:39:53 +00:00
orbiter
be0c492ae5 fix for memory leak bug in new dht transmissions 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5606 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-02-14 00:01:05 +00:00
hermens
2173865f92 Prevent race condition when switching timezones. 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5605 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-02-13 11:59:50 +00:00
orbiter
40d9849aa4 - better control of chunk size in dht selection 
- more restrict values in selection
- step to 4 vertical partitions

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5603 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-02-12 14:47:32 +00:00
orbiter
30a1de41b3 disabled the BufferedIOChunks, because I consider it as broken. 
I will try to fix that, but it is better to not use a buffer than using a broken buffer.

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5600 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-02-11 15:21:48 +00:00
orbiter
411f2212f2 more memory leak fixing hacks 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5599 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-02-11 13:31:10 +00:00
orbiter
985d421f91 found and fixed some memory leaks 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5596 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-02-11 11:24:15 +00:00
orbiter
333489420b - fix for NPE when loading the cytag image 
- some hacks for less memory usage:
-- less usage of buffer and cache memory in EcoFS
-- buffer allocation on-demand in BufferedIOChunks
-- removed largest ybr idx

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5595 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-02-11 10:52:56 +00:00
orbiter
6a32193916 - refactoring of cache naming in web index cache (no more dht semantics there) 
- activating a feature in the thread dump that cuts off dumping of a trance of inside-java-core events

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5593 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-02-10 23:28:13 +00:00
orbiter
6c627dbdff update to the server core 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5591 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-02-10 13:26:26 +00:00
orbiter
5393f356aa fix for termination problem 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5589 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-02-10 01:08:06 +00:00
orbiter
6a876ecb88 first fixes to the DHT transmission process 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5588 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-02-10 00:48:54 +00:00
orbiter
c25c334b75 replaced old DHT transmission method with new method. Many things have changed! some of them: 
- after a index selection is made, the index is splitted into its vertical components
- from differrent index selctions the splitted components can be accumulated before they are placed into the transmission queue
- each splitted chunk gets its own transmission thread
- multiple transmission threads are started concurrently
- the process can be monitored with the blocking queue servlet
To implement that, a new package de.anomic.yacy.dht was created. Some old files have been removed.
The new index distribution model using a vertical DHT was implemented. An abstraction of this model
is implemented in the new dht package as interface. The freeworld network has now a configuration
of two vertial partitions; sixteen partitions are planned and will be configured if the process is bug-free.
This modification has three main targets:
- enhance the DHT transmission speed
- with a vertical DHT, a search will speed up. With two partitions, two times. With sixteen, sixteen times.
- the vertical DHT will apply a semi-dht for URLs, and peers will receive a fraction of the overall URLs they received before.
  with two partitions, the fractions will be halve. With sixteen partitions, a 1/16 of the previous number of URLs.
BE CAREFULL, THIS IS A MAJOR CODE CHANGE, POSSIBLY FULL OF BUGS AND HARMFUL THINGS.

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5586 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-02-10 00:06:59 +00:00
orbiter
e9a4182e6a using a concurrent hash map for the template cache 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5584 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-02-08 21:48:37 +00:00
lotus
e8ae2599fd * some refactoring/moves to consoleInterface 
* added possibility to find maximum possible heap size
you can get it via getWin32MaxHeap.bat
this may cause high system load
moreover the found limit is no guarantee for stable startups since it depends on system configuration

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5583 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-02-07 11:53:48 +00:00
orbiter
01b97ef3f8 added new cybertag-tracking feature that was inspired by itgrl 
from the forum discussion in
http://forum.yacy-websuche.de/viewtopic.php?p=12612#p12612

The feature will provide two basic entities:
- you can integrate image links which point to your yacy installation anywhere in the web.
  the image can be loaded with
  <img src="http://<yourpeer>:<yourport>/cytag.png?icon=invisible&nick=<yournickname_or_community_id>&tag=<anything>">
  This will place a invisible 1-pixel image. If you change the icon=invisible to icon=redpill, you will see a red pill
  Use this, to track your activity in the web.
- you can view your tracks at
  http://localhost:8080/Tracks.html
- There is a public api to your tracks at
  http://localhost:8080/api/tracks_p.json
  which needs authentication


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5581 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-02-06 15:06:19 +00:00
borg-0300
b19bc611b0 gc: better logging 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5578 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-02-05 19:42:32 +00:00
orbiter
b1f9c00118 fix for bug in merge operator initialization 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5577 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-02-05 15:26:16 +00:00
orbiter
b57c9da1f8 - fixes to doc, ppt, xls parser: better title 
- fixes to httpd server response header generation
- fixes to a server date computation bug
- new Button in indexControl to view content of url in ViewFile


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5576 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-02-05 15:15:13 +00:00
f1ori
7936e58fe7 * sorry,previous version didn't compile 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5575 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-02-05 12:15:21 +00:00
f1ori
76cdc59789 * added some convertions to and from UTF-8 
* this might fix problems on windows systems
  (like http://forum.yacy-websuche.de/viewtopic.php?f=6&t=1824)


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5574 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-02-05 12:12:07 +00:00
orbiter
65a1de6c05 longer timeout for remote crawl queries 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5573 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-02-04 16:55:09 +00:00
orbiter
072dd01ac8 more logging for RSS parser (to fix the remote crawl problem) 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5572 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-02-04 16:50:10 +00:00
orbiter
9d282d2c16 - renamed interactivesearch to yacyinteractive 
- added a configuration option to set the pop up page in Config Appearance
- added a minimized header option to yacyinteractive
- fixed a bug in yacysearch: default values when no query is done


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5569 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-02-03 13:04:02 +00:00
orbiter
180fe81ef7 quick hack to copy new log configuration over old one 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5565 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-02-02 14:49:46 +00:00
orbiter
d3e33fd6c1 removed strange retry logic from DHT transfer 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5564 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-02-01 23:02:40 +00:00
orbiter
db510b5d52 more exception logging 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5561 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-02-01 22:05:09 +00:00
orbiter
ef82cced01 removed default line 'P2P WEB SEARCH' if no line is given 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5553 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-02-01 00:43:52 +00:00
orbiter
78b7361937 fixed problem with logging 
YOU MUST DELETE DATA/LOG TO MAKE THIS WORK! (sorry..)

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5552 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-02-01 00:21:08 +00:00
low012
f136ddcfd4 *) this change is supposed to prevent the creation of temporary files by Apache Commons Fileupload library in cases where it is not necessary (as proposed by thq in http://forum.yacy-websuche.de/viewtopic.php?f=8&t=1806) 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5546 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-01-31 09:17:58 +00:00
orbiter
94110df85a moved logging partially to kelondro 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5545 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-01-31 01:06:56 +00:00
orbiter
024da2916b refactoring of logging 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5544 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-01-30 23:33:47 +00:00
orbiter
83ce65707a (almost) completed partition of classes in kelondro 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5543 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-01-30 22:44:20 +00:00
orbiter
7ee494fde5 more refactoring of kelondro: 
- seperated BLOB from table classes
- renamed 'coding' package to 'order'

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5542 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-01-30 22:08:08 +00:00
lotus
d4281b78da dynamic memory scale 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5541 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-01-30 18:53:51 +00:00
orbiter
bf93767ec6 refactoring of kelondro database classes 
(to be continued)

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5540 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-01-30 15:33:00 +00:00
orbiter
fc27bf8c4c refactoring of kelondro classes: 
kelondro shall become independent from other packages.
moved bytebuffer, date and memory to kelondro

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5539 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-01-30 14:48:11 +00:00
orbiter
fe77fc3d62 - added new property setting 'repositoryPath' 
which can be used to map any path to http://localhost:8080/repository/
  This can be used to do an intranet-indexing without the setting of
  symbolic links - which does not work in Windows environment.
  Now also Windows users can index their file system easily
  using the intranet use case.
- fixed some problems with the identification of the alternative
  path in DATA/HTDOCS in the httpd file server

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5538 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-01-30 13:30:36 +00:00
orbiter
6cbca1e508 extended last fix, preventing more sorts 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5533 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-01-29 16:42:01 +00:00
orbiter
f9672d3f97 applied fix for inefficient put method as recommended by celle, see 
http://forum.yacy-websuche.de/viewtopic.php?p=12424#p12424

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5532 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-01-29 16:08:24 +00:00
apfelmaennchen
3484e55be4 - small fix for bookmarksDB 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5527 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-01-28 06:57:11 +00:00
apfelmaennchen
6dd52422ea - added two dialogs to manage bookmark tags in YaCy-UI 
- fixed renameTag() in bookmarksDB
- added /api/bookmarks/tags/addTag.xml
- added /api/bookmarks/tags/editTag.xml

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5525 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-01-28 00:15:43 +00:00
orbiter
3154926311 some better memory protection and OOM prevention in EcoFS 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5523 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-01-26 20:29:20 +00:00
f1ori
aaafe05c02 * revert debug change 
* contains instead of startsWith, because there might me localizied strings
* decode punycode for every domainpart seperately (see http://forum.yacy-websuche.de/viewtopic.php?f=9&t=1749)


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5516 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-01-24 00:33:38 +00:00
f1ori
5570fa817b * remove & from openBrowser command (see http://forum.yacy-websuche.de/viewtopic.php?f=5&t=1728&hilit=#p12321) 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5515 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-01-23 22:37:00 +00:00
orbiter
419469ac27 added more methods to control the vertical DHT (not yet active .. ) 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5514 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-01-23 15:32:27 +00:00
lotus
4ef6b15eb8 limit -Xmx setting to 1999m on win32. bigger values would never work. 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5513 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-01-23 15:13:09 +00:00
orbiter
dedfc7df7f removed distinction between DHT-in and DHT-out. This is necessary to make room for the new cell data structure, which cannot use this this distinction in the first place, but will enable the same meaning with different mechanisms (segments, later) 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5511 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-01-22 00:03:54 +00:00
orbiter
b74159feb8 preparations to integrate the new 'cell' index data structure 
(this commit is just to move development files to my other computer, no functionality change so far)

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5509 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-01-21 18:23:37 +00:00
orbiter
d399444e49 added debug information to class loader 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5508 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-01-21 18:20:15 +00:00
orbiter
5080fc33bf fix for http://forum.yacy-websuche.de/viewtopic.php?p=12247#p12247 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5506 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-01-20 22:57:58 +00:00
orbiter
335d6ce8fc fix for class loading problem 
see also http://forum.yacy-websuche.de/viewtopic.php?p=12153#p12153

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5505 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-01-20 22:47:58 +00:00
low012
78778df464 *) this should adjust the Dev/Main detection of the updater to the new version numbers (0.7x is Dev, if x != 0) 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5504 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-01-18 21:48:03 +00:00
orbiter
b423d0a036 moved all servlets from htroot/xml to htroot/api 
the file server contains a patch that temporary matches all xml paths to api,
that means all interfaces still work. Please adopt all your interfaces to the new path.

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5497 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-01-15 23:52:58 +00:00
orbiter
91af105373 last changes before release 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5493 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-01-13 23:49:08 +00:00
f1ori
7eade3f181 * fix for http://forum.yacy-websuche.de/viewtopic.php?f=5&t=1728 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5489 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-01-13 21:42:00 +00:00
orbiter
d1bace5e4d enhanced cleanup function 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5488 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-01-13 15:34:11 +00:00
orbiter
cb76d9e0e4 more synchronized in BLOBHeap (will not fix problem with Runtime-Error as reported in forum) 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5487 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-01-13 13:22:29 +00:00
orbiter
ff41da613e removed exception printout during load of snippets 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5484 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-01-13 00:30:19 +00:00
orbiter
814a28775f removed thread dump writing in case of invocation target exception in httpd (looked bad, not serious) 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5483 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-01-13 00:27:01 +00:00
orbiter
bed38a5f8c fix for uncaught exception in RSSReader 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5482 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-01-13 00:20:37 +00:00
orbiter
05c235de32 fix for npe 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5481 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-01-13 00:10:42 +00:00
low012
7608944081 *) bugfix for REMOTE_HOST environment variable in CGI code (shows hostname of client instead of hostname of YaCy peer now) 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5480 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-01-12 22:15:00 +00:00
orbiter
a6b29cf72c reverted change of search event processing in SVN 5460. The new code did not work properly, 
it gave remote search requests too less time


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5479 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-01-12 15:06:22 +00:00
orbiter
9ef77d57f5 added an access control to the search interface using white/blacklists: 
in the network configuration, you can configure a whiteliste and a blacklist
- blacklistet clients cannot search
- whitelistet client get never any search restrictions
- for all other clients: apply DoS search restrictions
Please see the example configuriation in yacy.network.freeworld.unit
by default, all clients from localhosts get whitlistet.
If you have your own YaCy network, please put all the IPs of your peers into the whitelist

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5475 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-01-12 10:55:48 +00:00
orbiter
efe801173c better dht-in cache flush. see also: 
http://forum.yacy-websuche.de/viewtopic.php?p=11936#p11936

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5472 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-01-11 22:39:49 +00:00
orbiter
941ab78d9b better termination for blocking threads 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5471 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-01-11 22:34:08 +00:00
apfelmaennchen
3dc208fad0 bugfix: bookmarks can now handle folder names like /news and /newspaper without getting confused... 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5470 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-01-11 19:39:51 +00:00
orbiter
e948df68ac longer timeout for queues during shutdown 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5469 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-01-11 19:10:09 +00:00
orbiter
2b32248079 fix for http://forum.yacy-websuche.de/viewtopic.php?f=6&t=1516&p=10545#p10545 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5468 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-01-11 19:02:50 +00:00
low012
c1330f5743 *) added environment variable DOCUMENT_ROOT 
*) caught exception

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5466 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-01-10 18:31:10 +00:00
low012
f26b8fcb1b *) comment mode is 'moderated' instead of 'activated' by default now (to avoid spam being visible) 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5465 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-01-10 12:58:35 +00:00
orbiter
b2a8c653ee small fixes 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5464 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-01-10 09:21:44 +00:00
orbiter
f675d47f86 better protection against database failures 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5463 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-01-10 09:19:46 +00:00
orbiter
4f45605f04 small update for timing in search result processing 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5460 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-01-09 15:28:45 +00:00
orbiter
9d119c6b61 migration of auto-update rules to new release strategy: 
next stable will be 0.7, development releases are 0.*x, experimental will be if x = 1, 2, 3

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5458 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-01-09 10:08:11 +00:00
orbiter
4d5b401f00 try to fix some performance problems with the internal index management: 
- ensuring that ordered indexes stay ordered during remove
- no unnecessary ordering checks
- better test logic in crawl stacker

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5457 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-01-09 00:06:36 +00:00
borg-0300
a0605325bb fixed a NullPointer Exception 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5452 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-01-07 15:44:09 +00:00
orbiter
b2b7edae18 fixed interactive search 
- added dummy servlet class, because otherwise the template engine is not triggered.
thats so because the yacy httpd works much faster as normal file server without a scan
of the served pages. Therefore each page with templates must now have a class file associated to it.
- fixed json output format of yacysearch

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5449 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-01-06 20:04:09 +00:00
lotus
2be119f0df adjusted big peer to 28M links 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5448 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-01-06 18:20:06 +00:00
orbiter
c6880ce28b removed the permanent cache flush and replaced it with a periodic cache flush 
The cache is now flushed only for one second every ten seconds. During a crawl the cache
fills up completely, and is only flushed if space is needed for more documents.

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5446 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-01-06 13:51:59 +00:00
orbiter
ef7fe537c5 fixed a cache-bug in cachedFileRA 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5445 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-01-06 10:51:56 +00:00
orbiter
6c7e83909b - refactoring of data access methods to be prepared for new cell data structure 
- removed a memory overhead in collections which prevent OOM Exception in low memory configurations

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5443 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-01-06 09:38:08 +00:00
lotus
c8451614f3 fix for overflow 
http://forum.yacy-websuche.de/viewtopic.php?p=11696#p11696

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5440 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-01-05 18:28:27 +00:00
orbiter
c4c4c223b9 fixed a problem with attribute flags on RWI entries that prevented proper selection of index-of constraint 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5437 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-01-04 02:27:29 +00:00
orbiter
6072831235 no cr transmission for robinson peers 
see also: http://forum.yacy-websuche.de/viewtopic.php?p=10290#p10290

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5436 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-01-03 23:44:42 +00:00
low012
afe98bc11c *) added changes as proposed by Halborinda in http://forum.yacy-websuche.de/viewtopic.php?f=6&t=1674 
*) changed indention

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5434 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-01-03 08:24:08 +00:00
orbiter
07fc115e90 removed active profiling in kelondroRowSet 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5433 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-01-02 12:33:06 +00:00
orbiter
be4c458951 refactoring (implemented Iterable in kelondroRowCollection) 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5432 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-01-02 11:38:20 +00:00
low012
bb5c2cd12e *) ISINDEX parameters will not be put on commandline anymore to prevent possible security hazards (better safe than sorry). Parmeters will have to be read from QUERY_STRING in ISINDEX case too which does not seem to be uncommon behaviour for web servers: http://vms.pdv-systeme.de/users/martinv/cgi_basics/cgi_basics.html#Datenuebergabe 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5431 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-01-02 11:18:26 +00:00
orbiter
b6bba18c37 replaced the storing procedure for the index ram cache with a method that generates BLOBHeap-compatible dumps 
this is a migration step to support a new method to store the web index, which will also based on the same data structure. made also a lot of refactoring for a better structuring of the BLOBHeap class.

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5430 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-01-01 22:31:16 +00:00
low012
db1cfae3e7 *) cleaning up after myself 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5429 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-01-01 19:45:15 +00:00
low012
f547f9a78c *) added CGI capabilities (run Perl scripts and other software via HTTP GET and POST) 
*) set cgi.allow to true in yacy.conf to enable CGI (CGI is disabled by default)
*) edit cgi.suffixes in yacy.conf if necessary to use additional script types

ATTENTION: This is a rather experimental feature, not all environment variables are set yet. 

Only enable CGI if you know what you are doing. Poorly implemented CGI scripts can put a system's integrity at risk!

Implementation of more environment variables and documentation due for the next days.

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5428 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-01-01 19:40:06 +00:00
f1ori
bdc380cd84 * add lastModified to templateCache 
-> no outdated files from cache anymore...


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5427 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2009-01-01 14:56:53 +00:00
f1ori
025094675f * remove empty directory 
* add necessary dependency for pdfParser


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5424 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-12-31 19:39:02 +00:00
f1ori
c5691180cb * skip style-tags in HTML-files 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5423 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-12-31 19:34:24 +00:00
orbiter
3567c58b18 added another filed information for BLOBHeap dumps: the gaps 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5420 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-12-30 10:49:43 +00:00
orbiter
abdd4aa414 added a index dump for blob heaps: 
this will increase the shutdown time for at most some seconds, but will speed up the start-up

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5419 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-12-29 21:36:27 +00:00
orbiter
8c3205b62e fix for OOB Exception 
see http://forum.yacy-websuche.de/viewtopic.php?p=11598#p11598

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5417 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-12-29 17:36:53 +00:00
orbiter
78c568331e added test channel to /xml/feed.rss 
can be obtained with 
http://localhost:8080/xml/feed.rss?set=TEST
returns always a single feed entry with a fresh date

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5416 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-12-29 12:39:07 +00:00
orbiter
e004da48d3 - added fast fingerprint computation for files (any). Will be used in new index dump method 
- refactoring

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5415 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-12-29 12:22:13 +00:00
f1ori
2d2ce24011 * remove all encoding-stuff from proxy 
encoding is handled by parsers or browser, proxy only passes through


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5410 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-12-23 19:14:54 +00:00
f1ori
73c8a0839c * abort download, when proxy connection is closed 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5409 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-12-23 11:30:24 +00:00
orbiter
bb935fdbb0 less organization overhead for DNS caching and prefetching 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5408 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-12-23 10:06:49 +00:00
f1ori
4907697cfa * make fileuploads through proxy bigger than 65500 bytes possible 
* remove gzip-encoding for files from cache


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5407 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-12-22 23:04:00 +00:00
orbiter
fc8189f3fb better self-healing of corrupted databases 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5406 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-12-22 16:43:49 +00:00
f1ori
963da8c3f9 * updated tm-extractors to new version 1.0 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5405 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-12-21 14:51:03 +00:00
orbiter
e34ac22fbd - added new monitoring servlet at 
http://localhost:8080/PerformanceConcurrency_p.html
- used the new monitoring to do some fine-tuning of the indexing queue

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5402 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-12-19 15:26:01 +00:00
lotus
449e697436 fix for null-seed in seedfile 
http://forum.yacy-websuche.de/viewtopic.php?f=5&t=1653

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5401 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-12-19 12:10:01 +00:00
orbiter
d376d81fc4 replaced busy thread control of crawl stacker by blocking threads 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5400 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-12-18 23:18:34 +00:00
orbiter
f29b48d9ff patch for IndexOutOfBoundsException 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5399 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-12-18 22:05:26 +00:00
f1ori
0881190b19 * Robots.txt: don't interpret Crawl-Delays for other robots 
fixes: http://forum.yacy-websuche.de/viewtopic.php?f=5&t=1647


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5398 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-12-18 15:35:41 +00:00
orbiter
243e73f53b removed unnecessary usage of kelondroBLOBTree 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5397 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-12-18 00:18:37 +00:00
orbiter
8cb7170b75 - set status of kelondroTree, kelondroBLOBTree and kelondroFlexTable to deprecated 
- removed initialization and/or usage of kelondroFlexTable (should meanwhile not be used any more)

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5396 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-12-18 00:08:17 +00:00
orbiter
7535fd7447 - refactoring of CrawlEntry and CrawlStacker 
- introduced blocking queues in CrawlStacker to make it ready for concurrency
- added a second busy thread for the CrawlStacker
The CrawlStacker is multithreaded. It shall be transformed into a BlockingThread in another step.
The concurrency of the stacker will hopefully solve some problems with cases where DNS blocks.

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5395 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-12-17 22:53:06 +00:00
lotus
18513e2ee2 npe fix: http://forum.yacy-websuche.de/viewtopic.php?t=1646 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5393 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-12-16 13:36:13 +00:00
orbiter
2802138787 - refactoring of CrawlStacker (to prepare it for new multi-Threading to remove DNS lookup bottleneck) 
- fix of shallBeOwnWord target computation heuristic


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5392 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-12-15 00:02:58 +00:00
orbiter
db6b3bf5a3 speed enhancement for integrated http server: 
- tuning hacks in template engine
- bypassing the template engine if no servlet present

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5389 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-12-11 20:10:37 +00:00
orbiter
7cd08bd5fb fix for NPE in BLOBCompressor 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5388 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-12-11 13:33:24 +00:00
orbiter
5b94498643 fine-tuning of cache usage from SVN 5386 and a bug fix for overflow in available() method 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5387 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-12-10 14:35:01 +00:00
orbiter
1779c3c507 - added a read cache to the RAFile interface to RandomAccessFile 
- added a write buffer to BLOBHeap
- modified the BLOBBuffer (is now only to buffer non-compressed content)
- added content compression to the HTCache
The new read cache will decrease the start/initialization time of BLOB files,
like the HTCache, RobotsTxt and other BLOBHeap structures.


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5386 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-12-10 11:15:19 +00:00
orbiter
e1acdb952c fix for problem with userDB and bookmarksDB which was caused by changes in kelondroRA in SVN 5376 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5385 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-12-08 00:17:45 +00:00
orbiter
4a2dac659e more speed hacks: 
- modified and activated write buffer
- increased cache flush factor
- fixed a problem with deadlocking of indexing process

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5382 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-12-05 13:55:48 +00:00
orbiter
47292e696a more performance hacks 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5379 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-12-04 12:54:16 +00:00
orbiter
759cef23dd fix for bug in kelondroAbstractRA.readFully 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5378 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-12-03 23:32:07 +00:00
orbiter
d39d420b39 performance hacks 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5376 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-12-03 15:38:29 +00:00
orbiter
0b4808ba3d added new interactive search feature: 
- during the user types search queries, the local database is searched
- results are presented interactively

This was implemented using a new JSON result format for search results in YaCy
- added JSON as file format for servlets
- refactoring of current search servlets (xml and html)
- added JSON output format for search results
- added AJAX-based search page, that uses the yacysearch.json selrvlet to print results as a query is typed

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5373 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-12-02 15:24:25 +00:00
orbiter
74a3d86114 fixed a error response that might present classified information 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5372 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-12-01 23:14:42 +00:00
orbiter
c6525ab75f fix for NPE in seed handling 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5371 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-12-01 23:08:27 +00:00
lotus
1951d30a62 addendum to last commit 
handle words with length < 3 correctly

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5369 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-11-26 19:43:40 +00:00
lotus
325ba7bfb8 only query words with length > 2 
this is not complete, yet

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5368 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-11-26 16:41:38 +00:00
low012
e423fa9846 *) added method to only get file names in directory listing which match a filter 
*) only files which end with .black will be listed as blacklists
*) added a little bit of Javadoc

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5366 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-11-25 20:26:06 +00:00
orbiter
513179f404 changed interface to colletctionIndex and adopted all implementing classes: 
do not return a result of a double-check when adding entries with addUnique

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5363 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-11-23 23:55:08 +00:00
orbiter
9d64693cfb reverting again the changes to new concurrent chunkIterator 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5362 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-11-23 22:22:44 +00:00
orbiter
45ad1c3dd5 - re-activated concurrent iterator for EcoFiles 
- added javadoc for new concurrent intialization in kelondroBytesLongMap
- switched default value for commons storage to false
- version step

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5361 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-11-23 18:25:40 +00:00
orbiter
2e2120046f speed enhancement for BLOBHeap opening process 
using concurrency of FileIO and content processing

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5360 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-11-23 17:38:01 +00:00
orbiter
fa26a8f25a fix for deadlock-like behavior in balancer 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5358 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-11-22 11:25:01 +00:00
orbiter
1918a0173e added more exception handling during crawling 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5357 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-11-22 00:40:18 +00:00
orbiter
10f5ec1040 reverted last commit (more testing needed) 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5356 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-11-22 00:12:50 +00:00
f1ori
5af8923f37 * distribute forgotten jar-file in parser 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5355 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-11-22 00:05:04 +00:00
orbiter
b0f2003792 fast database initialization and fast start.up of yacy: 
- applied knowledge about concurrent files stream reading and index processing from the wikimedia reader
   to the EcoTable initialization process: the file reader is now concurrent to the index generation
- changed also some initialization processes to avoid some pauses during initialization

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5354 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-11-21 23:21:33 +00:00
orbiter
0ca4bc7b79 - added reader and visualization for mediawiki-export files: 
files exported from mediawiki using the xml schema according to
http://www.mediawiki.org/xml/export-0.3/
can be processed to be viewed in a YaCy servlet.
To acces such a file, place it into
DATA/HTCACHE/mediawiki/
i.e. the export from german wikipedia would be:
DATA/HTCACHE/mediawiki/wikipedia.de.xml
This file can then be accessed using the URL
http://localhost:8080/mediawiki_p.html?dump=wikipedia.de.xml&title=YaCy
if this is done the first time, an index file is created
(for this case: more than 4 million lines must be written, this takes about 15 minutes)
Then try the same url again.

- enhanced also the md5 computation speed


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5352 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-11-20 18:31:52 +00:00
danielr
2e63f03ca5 copy&paste vergessen :/ 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5351 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-11-20 11:41:11 +00:00
danielr
cd8082b4e3 fix for http://forum.yacy-websuche.de/viewtopic.php?f=5&t=1111#p11166 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5350 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-11-20 11:18:19 +00:00
lotus
4f996a7651 fix for logparser pattern 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5349 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-11-17 16:23:17 +00:00
f1ori
d18c18971e * dirlisting in UTF-8 encoding 
* fix for http://forum.yacy-websuche.de/viewtopic.php?f=6&t=1550&hilit=#p11108


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5348 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-11-15 20:49:03 +00:00
orbiter
867d0f2f56 removed some unnecessary pause delays 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5346 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-11-14 23:36:33 +00:00
f1ori
d49ffcd818 * files distributed by yacy are utf-8, files from repository use the system default charset 
* fixes http://forum.yacy-websuche.de/viewtopic.php?f=6&t=1564#p11092
  and http://forum.yacy-websuche.de/viewtopic.php?f=6&t=1550


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5345 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-11-14 20:49:16 +00:00
orbiter
8c96bc2ac1 do not use proxy caching rules for crawling 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5344 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2008-11-14 16:31:04 +00:00