Marown/yacy_search_server
1
0
Fork 0
mirror of https://github.com/yacy/yacy_search_server.git synced 2024-09-21 00:00:13 +02:00
Code Issues Actions Packages Projects Releases Wiki Activity
385 Commits 1 Branch 20 Tags 836 MiB
9f505af7aa
Commit Graph

137 Commits

Author SHA1 Message Date
orbiter
9f505af7aa preparations for bulk remote crawls 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@408 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-07-19 00:26:31 +00:00
orbiter
9c72b4cdec replaced index dump stack by an dump array and limited url number in assortment ram (prevents too much RAM occupation) 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@406 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-07-18 13:32:44 +00:00
orbiter
51962d55bf added 'PPM', page-per-minute statistics 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@405 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-07-18 00:44:51 +00:00
orbiter
159f795f65 bugfix (null pointer exception in assortments) 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@404 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-07-17 22:25:50 +00:00
orbiter
1d2155675b changed assortment memory cache flush 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@403 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-07-17 21:22:18 +00:00
orbiter
19dbed7cc8 code clean-up 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@401 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-07-12 15:09:35 +00:00
orbiter
00f63ea00d fail-save patch for pattern matching 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@400 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-07-12 00:31:35 +00:00
orbiter
0a6be961ea added pattern organization 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@399 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-07-12 00:21:46 +00:00
orbiter
40036ba69c fixed dht transmission; added url-blacklist blocking also for remote search 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@398 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-07-12 00:07:09 +00:00
orbiter
311e627363 blocking of blacklisted urls in indexReceive and small changes 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@397 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-07-11 15:36:10 +00:00
orbiter
2f0d7ea8d3 removed htcache stati (superfluous now) 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@396 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-07-09 00:33:34 +00:00
orbiter
277048501e bugfix 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@395 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-07-08 16:24:07 +00:00
orbiter
8b89c46afe fixed problem with cache write 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@394 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-07-08 16:06:35 +00:00
orbiter
455ae9f55f fixed htcache-store problem and due-time for remote crawls 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@393 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-07-08 15:17:50 +00:00
theli
55d10b864c *) further improvements in shutdown behaviour 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@392 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-07-07 22:03:52 +00:00
orbiter
419f8fb398 fixed bugs/missing code regarding new crawl stack 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@384 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-07-07 01:38:49 +00:00
orbiter
112c5d3332 the new file-based indexing queue 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@382 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-07-06 14:50:01 +00:00
orbiter
858cd94299 replaced indexing ram-queue by file-based stack-queue 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@381 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-07-06 14:48:41 +00:00
theli
57c30f1d78 *) bugfix for usage of httpc without gzip content encoding 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@369 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-07-04 11:25:25 +00:00
theli
0e2c33ee55 *) Network.html/Network.java: 
- Adding function to manually force peer ping to remote yacy peer
  See:Network.html?page=4
- for debugging purpose only!

*) serverAbstractThread.java:
- Adding posibility to notify a server thread via a synchronization object
- this is needed e.g. by the port forwarding feature to send a notification
  to the peerPing thread to redo peer-ping with the new ip/port Settings_p.html

*) Port Forwarding Feature (it should work now)
- adding a serverThread which is responsible to detect broken port forwarding 
  connections and to do reconnect if needed
- serverCore.java: moving port forwarding initialization into a separate function
- adding positility to configure the ssh port 
- moving configuration section on the gui into a separate fieldset
- hello.java: only trying to do a second connect to the clientIp address during
  peer handshake if either remote port forwarding is not enabled locally or
  the clientIP is not equal to any local ip

*) httpdFileHandler.java:
- printout a more verbose errormessage

*) httpc.java
- allowing to deactivate content encoding from outside


 

*) plasmaCrawlWorker.java
- the crawler worker now tries to refetch the content of a website without
  gzip content encoding if a gzip error occured



git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@368 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-07-04 11:09:48 +00:00
orbiter
5159a090b0 fixed parser bug with lowercase force (appeared in: http://spellbound.sourceforge.net/) 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@367 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-07-03 23:33:25 +00:00
orbiter
7f7cbc5019 fixed bug with snippets 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@365 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-07-03 13:37:20 +00:00
orbiter
eb74fa0c82 fixed a bug with snippet-length 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@359 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-07-01 23:35:36 +00:00
orbiter
86f2aa8478 fixed seed-load date bug (evaluating server date for age computation) 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@354 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-06-30 23:19:08 +00:00
orbiter
664bceced5 removed debug-lines 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@351 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-06-30 18:56:01 +00:00
orbiter
75ebdbc852 enhanced snippet-generation (case where snippet is too long) 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@350 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-06-30 18:54:00 +00:00
orbiter
8a4f297324 fixed/enhanced snippet error-handling; suppression of results where no snippet exists 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@347 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-06-30 00:01:53 +00:00
orbiter
712fe9ef18 bugfixed utf-8 decoding and parser 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@346 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-06-29 22:55:37 +00:00
theli
eee6322aaf *) Adding redirection support to plasmaCrawlWorker.java 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@328 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-06-28 08:07:41 +00:00
theli
cd279907c0 *) Adding redirection support to plasmaCrawlWorker.java 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@327 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-06-28 08:01:26 +00:00
theli
6697d5e52e *) correcting fkt. mediaExtContains 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@326 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-06-28 06:44:31 +00:00
orbiter
3addf58046 enhanced snippet-loading with threads 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@322 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-06-24 07:41:07 +00:00
orbiter
56d28a16f0 bugfixes 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@320 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-06-23 14:40:39 +00:00
orbiter
d6c85228a6 enhanced snippet computation 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@319 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-06-23 12:12:12 +00:00
theli
fafda068f9 *) allowing crawler to process resources with statuscode 203 
- this is needed if yacy is behind a second proxy 

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@316 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-06-23 10:00:31 +00:00
theli
aae9a433a6 *) correcting usage of supportedFileExt-List 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@315 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-06-23 07:43:59 +00:00
orbiter
1e7f062350 many bugfixes, memory leak fixes, performance enhancements; new kelondroHashtable; activated snippets 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@313 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-06-23 02:07:45 +00:00
orbiter
68dc2b0c6b added kelondroArray, the basis for upcoming kelondroHash and some bug fixes 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@311 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-06-21 01:17:25 +00:00
orbiter
a19541e563 code-enhancements after analysis with AppPerfect 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@307 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-06-20 16:36:31 +00:00
orbiter
85075269a6 extended fail-safe memory-managament. prevents too much allocation, too often GC and should help for the 100%CPU-bug 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@303 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-06-20 00:46:23 +00:00
orbiter
e3c92818db avoiding OutOfMemoryError routines 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@302 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-06-19 13:37:17 +00:00
orbiter
3e8ee5a46d enhanced caching in kelondroRecords and added better synchronization/finalizer 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@301 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-06-19 05:27:42 +00:00
theli
db3ed75728 *) closing stream correctly 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@293 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-06-17 07:58:02 +00:00
orbiter
5d06ded005 enhanced html parser speed 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@290 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-06-17 01:26:51 +00:00
orbiter
5a490aa065 fixed html parser 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@289 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-06-16 21:49:56 +00:00
orbiter
a25b5b4986 fixed possible memory leak in htmlScraper: be aware that now links can get lost; further work necessary 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@288 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-06-16 18:31:28 +00:00
theli
9e47ba5ad6 *) adding missing calls for function close() to avoid "too many open file" bug 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@282 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-06-16 08:34:52 +00:00
theli
9a98988c3c *) Bugfix for SSL/NIO Bug 
See: http://www.yacy-forum.de/viewtopic.php?t=516
   - removing NIO from server/serverCore.java because of massive problems
     with socket close issues
*) Adding support for remote port forwarding via sch
   @Orbiter: Please take a look into
   - hello.java
   - server/serverCore.java.publicIP()
   - yacy/yacyClient.java.publishMySeed(...)
*) Making startup loading of additional content parsers more failsafe


git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@281 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-06-16 07:28:07 +00:00
orbiter
a1ffc27041 preparations for image/movie/music indexing 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@280 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-06-16 00:31:13 +00:00
orbiter
a5b40923b6 added word migration to assortments (start with 'java -classpath classes yacy -migratewords') 
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@278 6c8d7289-2bf4-0310-a012-ef5d649a1542
 2005-06-15 01:22:07 +00:00