Marown/yacy_search_server
1
0
Fork 0
mirror of https://github.com/yacy/yacy_search_server.git synced 2024-09-19 00:01:41 +02:00
Code Issues Actions Packages Projects Releases Wiki Activity
add0e42804
yacy_search_server/source/net/yacy/search/Switchboard.java

3848 lines
182 KiB
Java
Raw Normal View History

fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
// plasmaSwitchboard.java
// (C) 2004-2007 by Michael Peter Christen; mc@yacy.net, Frankfurt a. M., Germany
// first published 2004 on http://yacy.net
//
// This is a part of YaCy, a peer-to-peer based web search engine
//
// $LastChangedDate$
// $LastChangedRevision$
// $LastChangedBy$
//
// LICENSE
//
// This program is free software; you can redistribute it and/or modify
// it under the terms of the GNU General Public License as published by
// the Free Software Foundation; either version 2 of the License, or
// (at your option) any later version.
//
// This program is distributed in the hope that it will be useful,
// but WITHOUT ANY WARRANTY; without even the implied warranty of
// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
// GNU General Public License for more details.
//
// You should have received a copy of the GNU General Public License
// along with this program; if not, write to the Free Software
// Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
/*
This class holds the run-time environment of the plasma
Search Engine. It's data forms a blackboard which can be used
to organize running jobs around the indexing algorithm.
The blackboard consist of the following entities:
- storage: one plasmaStore object with the url-based database
- configuration: initialized by properties once, then by external functions
- job queues: for parsing, condensing, indexing
*/
package net.yacy.search;
import java.io.BufferedInputStream;
import java.io.BufferedOutputStream;
import java.io.BufferedReader;
import java.io.ByteArrayInputStream;
import java.io.ByteArrayOutputStream;
import java.io.File;
import java.io.FileInputStream;
import java.io.FileNotFoundException;
import java.io.FileOutputStream;
import java.io.IOException;
import java.io.InputStream;
import java.io.OutputStream;
import java.io.Reader;
import java.net.MalformedURLException;
import java.security.NoSuchAlgorithmException;
import java.security.PublicKey;
import java.security.spec.InvalidKeySpecException;
import java.text.ParseException;
import java.text.SimpleDateFormat;
import java.util.ArrayList;
import java.util.Collection;
import java.util.Date;
import java.util.HashMap;
Enhanced crawl start for very, very large crawl lists (i.e. > 5000) which had a problem because of badly used concurrency. This fix also caused a redesign of the whole host deletion process. This should fix bug http://bugs.yacy.net/view.php?id=250
2013-10-24 16:20:20 +02:00
import java.util.HashSet;
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
import java.util.Iterator;
import java.util.LinkedHashSet;
import java.util.List;
import java.util.Map;
import java.util.Properties;
import java.util.Set;
import java.util.SortedMap;
import java.util.SortedSet;
import java.util.TreeMap;
import java.util.TreeSet;
import java.util.concurrent.ConcurrentHashMap;
import java.util.concurrent.ExecutionException;
import java.util.concurrent.LinkedBlockingQueue;
import java.util.concurrent.Semaphore;
import java.util.concurrent.atomic.AtomicInteger;
import java.util.regex.Pattern;
import java.util.regex.PatternSyntaxException;
import java.util.zip.GZIPInputStream;
import java.util.zip.GZIPOutputStream;
import java.util.zip.ZipEntry;
import java.util.zip.ZipInputStream;
purge a lucene cache - possible memory leak fix
2013-11-18 22:47:35 +01:00
import org.apache.lucene.search.FieldCache;
added the clickdepth post-processing: some links may have 'shortcuts' to already calculated click depths. There are then calculated if the crawl buffer is empty and therefore no new 'shortcuts' can be discovered. The status of the clickdepth stack (to-be-processed) can be seen using a solr search command like this: http://localhost:8090/solr/select?q=process_sxt:[*%20TO%20*]&start=0&rows=30&fl=sku,clickdepth_i,process_sxt
2013-01-04 16:37:39 +01:00
import org.apache.solr.common.SolrDocument;
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
import org.apache.solr.common.SolrInputDocument;
import net.yacy.contentcontrol.ContentControlFilterUpdateThread;
import net.yacy.contentcontrol.SMWListSyncThread;
import net.yacy.cora.date.GenericFormatter;
import net.yacy.cora.document.WordCache;
import net.yacy.cora.document.analysis.Classification;
- the webgraph shall store all links which appear on a web page and not all unique links! This made it necessary, that a large portion of the parser and link processing classes must be adopted to carry a different type of link collection which carry a property attribute which are attached to web anchors. - introduction of a new URL class, AnchorURL - the other url classes, DigestURI and MultiProtocolURI had been renamed and refactored to fit into a new document package schema, document.id - cleanup of net.yacy.cora.document package and refactoring
2013-09-15 00:30:23 +02:00
import net.yacy.cora.document.encoding.ASCII;
import net.yacy.cora.document.encoding.UTF8;
import net.yacy.cora.document.feed.RSSFeed;
import net.yacy.cora.document.feed.RSSMessage;
import net.yacy.cora.document.feed.RSSReader;
- replaced the properties object in AnchorURL with distinct variables for anchor attributes. - this caused that large portions of the parser code had to be adopted as well - added a counter target_order_i for anchor links in webgraph computation
2013-09-15 23:27:04 +02:00
import net.yacy.cora.document.id.AnchorURL;
- the webgraph shall store all links which appear on a web page and not all unique links! This made it necessary, that a large portion of the parser and link processing classes must be adopted to carry a different type of link collection which carry a property attribute which are attached to web anchors. - introduction of a new URL class, AnchorURL - the other url classes, DigestURI and MultiProtocolURI had been renamed and refactored to fit into a new document package schema, document.id - cleanup of net.yacy.cora.document package and refactoring
2013-09-15 00:30:23 +02:00
import net.yacy.cora.document.id.DigestURL;
import net.yacy.cora.document.id.MultiProtocolURL;
- removed ZURL data structure; removed also the ZURL data file - replaced load failure logging by information which is stored in Solr - fixed a bug with crawling of feeds: added must-match pattern application to feed urls to filter out such urls which shall not be in a wanted domain - delegatedURLs, which also used ZURLs are now temporary objects in memory
2013-09-17 15:27:02 +02:00
import net.yacy.cora.federate.solr.FailCategory;
changes in ranking computation - an existing ranking servlet for solr was extended. It is now possible to set boost values for fields, boost functions and boost queries. - The ranking can have different instances, but currently only the first one is used - added an abstraction layer for fields which can be used for search and those fields can be edited in the solr ranking configruation - the ranking value from solr within the field score is used to combine remote search requests, which all are created using the same locally defined boost values - reduced the number of fields which are used for search (makes it faster) - replaced some text fields by string fields (makes indexing faster) - removed classes which had no use - made a large number of experiments for a better ranking and created a temporary setting which prefers hits inside titles - adjusted also the RWI-based ranking computation to 'prefer title' - made special cases like for portal search where no post-processing and post-ranking is wanted: this keeps the original ranking order as done by Solr - fixed many bugs with old settings for ranking
2013-03-13 14:47:00 +01:00
import net.yacy.cora.federate.solr.Ranking;
introduced a second core named 'webgraph'. This core will hold the link structure, but is not filled yet. To have the opportunity of a second core, multi-core functionality had to be implemented to the deep-embedded solr: - migrated the solr_40 directory content to a subdirectory 'collection1'; the previously used default core is now called collection1 - added solr_40/webgraph subdirectory as second core - added a servlet configuration for the second core 'webgraph' in /IndexSchema_p.html - added instance handling as addition to solr connections: all solr connectors are now instances of an solr 'instance' object; this required a complete re-design of the solr embedding - migrated also caching and sharding ontop of new instance handling - migrated the search apis to handle now the access to a specific core, the default core named 'collection1' - migrated the remote solr search interface to access shards of cores; for the yacy remote search the default core is now called 'solr'; using the peer address as solr address - migrated the solr backup and restore process: old backups cannot be used after this migration! - redesign of solr instance handling in all methods which access the instances: they cannot hold copies of these instances any more; the must retrieve the actuall connection object every time they want to write to it (this solves also some bugs when switching the index/network) - added another schema 'solr.webgraph.schema', the old solr.keys.list is replaced by solr.collection.schema
2013-02-21 13:23:55 +01:00
import net.yacy.cora.federate.solr.SchemaConfiguration;
import net.yacy.cora.federate.solr.instance.RemoteInstance;
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
import net.yacy.cora.federate.yacy.CacheStrategy;
import net.yacy.cora.lod.JenaTripleStore;
import net.yacy.cora.order.Base64Order;
import net.yacy.cora.order.Digest;
import net.yacy.cora.order.NaturalOrder;
import net.yacy.cora.protocol.ClientIdentification;
import net.yacy.cora.protocol.ConnectionInfo;
import net.yacy.cora.protocol.Domains;
import net.yacy.cora.protocol.HeaderFramework;
import net.yacy.cora.protocol.RequestHeader;
import net.yacy.cora.protocol.ResponseHeader;
import net.yacy.cora.protocol.TimeoutRequest;
ftp client shall be able to open non-anonymous ftp servers if login details are given
2013-12-28 22:42:02 +01:00
import net.yacy.cora.protocol.ftp.FTPClient;
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
import net.yacy.cora.protocol.http.HTTPClient;
import net.yacy.cora.protocol.http.ProxySettings;
- refactoring of log to ConcurrentLog: jdk-based logger tend to block at java.util.logging.Logger.log(Logger.java:476) in concurrent environments. This makes logging a main performance issue. To overcome this problem, this is a add-on to jdk logging to put log entries on a concurrent message queue and log the messages one by one using a separate process. - FTPClient uses the concurrent logging instead of the log4j logger
2013-07-09 14:28:25 +02:00
import net.yacy.cora.util.ConcurrentLog;
logging of deadlocks (if any) during cleanup process
2013-07-17 12:38:58 +02:00
import net.yacy.cora.util.Memory;
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
import net.yacy.crawler.CrawlStacker;
import net.yacy.crawler.CrawlSwitchboard;
import net.yacy.crawler.HarvestProcess;
import net.yacy.crawler.data.Cache;
import net.yacy.crawler.data.CrawlProfile;
import net.yacy.crawler.data.CrawlQueues;
import net.yacy.crawler.data.NoticedURL;
import net.yacy.crawler.data.ResultImages;
import net.yacy.crawler.data.ResultURLs;
import net.yacy.crawler.data.NoticedURL.StackType;
import net.yacy.crawler.data.ResultURLs.EventOrigin;
import net.yacy.crawler.retrieval.Request;
import net.yacy.crawler.retrieval.Response;
import net.yacy.crawler.robots.RobotsTxt;
import net.yacy.data.BlogBoard;
import net.yacy.data.BlogBoardComments;
import net.yacy.data.BookmarkHelper;
import net.yacy.data.BookmarksDB;
import net.yacy.data.ListManager;
import net.yacy.data.MessageBoard;
import net.yacy.data.UserDB;
import net.yacy.data.WorkTables;
import net.yacy.data.wiki.WikiBoard;
import net.yacy.data.wiki.WikiCode;
import net.yacy.data.wiki.WikiParser;
import net.yacy.data.ymark.YMarkTables;
import net.yacy.document.Condenser;
import net.yacy.document.Document;
import net.yacy.document.LibraryProvider;
import net.yacy.document.Parser;
import net.yacy.document.TextParser;
import net.yacy.document.Parser.Failure;
import net.yacy.document.content.DCEntry;
import net.yacy.document.content.SurrogateReader;
import net.yacy.document.importer.OAIListFriendsLoader;
import net.yacy.document.parser.audioTagParser;
getting the trash out
2013-05-29 16:09:05 +02:00
import net.yacy.document.parser.pdfParser;
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
import net.yacy.document.parser.html.Evaluation;
import net.yacy.gui.Tray;
import net.yacy.kelondro.blob.Tables;
import net.yacy.kelondro.data.meta.URIMetadataNode;
import net.yacy.kelondro.data.word.Word;
Automatically flush the log cache if a short memory status is reached. For the default of 200 lines this can flush about 10MB.
2013-10-24 17:39:50 +02:00
import net.yacy.kelondro.logging.GuiHandler;
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
import net.yacy.kelondro.rwi.ReferenceContainer;
import net.yacy.kelondro.util.FileUtils;
import net.yacy.kelondro.util.MemoryControl;
import net.yacy.kelondro.util.OS;
import net.yacy.kelondro.util.SetTools;
import net.yacy.kelondro.workflow.BusyThread;
import net.yacy.kelondro.workflow.InstantBusyThread;
import net.yacy.kelondro.workflow.WorkflowProcessor;
import net.yacy.kelondro.workflow.WorkflowThread;
import net.yacy.peers.Dispatcher;
import net.yacy.peers.EventChannel;
import net.yacy.peers.Network;
import net.yacy.peers.NewsPool;
import net.yacy.peers.DHTSelection;
import net.yacy.peers.Protocol;
import net.yacy.peers.Seed;
import net.yacy.peers.SeedDB;
import net.yacy.peers.graphics.WebStructureGraph;
import net.yacy.peers.operation.yacyBuildProperties;
import net.yacy.peers.operation.yacyRelease;
import net.yacy.peers.operation.yacyUpdateLocation;
import net.yacy.repository.Blacklist;
import net.yacy.repository.Blacklist.BlacklistType;
import net.yacy.repository.FilterEngine;
import net.yacy.repository.LoaderDispatcher;
fixed and enhanced postprocessing
2013-11-16 08:23:21 +01:00
import net.yacy.search.index.Fulltext;
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
import net.yacy.search.index.Segment;
enhanced and fixed postprocessing
2013-11-15 15:41:12 +01:00
import net.yacy.search.index.Segment.ClickdepthCache;
import net.yacy.search.index.Segment.ReferenceReportCache;
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
import net.yacy.search.query.AccessTracker;
import net.yacy.search.query.SearchEvent;
import net.yacy.search.query.SearchEventCache;
import net.yacy.search.ranking.RankingProfile;
introduced a second core named 'webgraph'. This core will hold the link structure, but is not filled yet. To have the opportunity of a second core, multi-core functionality had to be implemented to the deep-embedded solr: - migrated the solr_40 directory content to a subdirectory 'collection1'; the previously used default core is now called collection1 - added solr_40/webgraph subdirectory as second core - added a servlet configuration for the second core 'webgraph' in /IndexSchema_p.html - added instance handling as addition to solr connections: all solr connectors are now instances of an solr 'instance' object; this required a complete re-design of the solr embedding - migrated also caching and sharding ontop of new instance handling - migrated the search apis to handle now the access to a specific core, the default core named 'collection1' - migrated the remote solr search interface to access shards of cores; for the yacy remote search the default core is now called 'solr'; using the peer address as solr address - migrated the solr backup and restore process: old backups cannot be used after this migration! - redesign of solr instance handling in all methods which access the instances: they cannot hold copies of these instances any more; the must retrieve the actuall connection object every time they want to write to it (this solves also some bugs when switching the index/network) - added another schema 'solr.webgraph.schema', the old solr.keys.list is replaced by solr.collection.schema
2013-02-21 13:23:55 +01:00
import net.yacy.search.schema.CollectionConfiguration;
import net.yacy.search.schema.CollectionSchema;
import net.yacy.search.schema.WebgraphConfiguration;
fixed and enhanced postprocessing
2013-11-16 08:23:21 +01:00
import net.yacy.search.schema.WebgraphSchema;
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
import net.yacy.server.serverCore;
import net.yacy.server.serverSwitch;
import net.yacy.server.http.RobotsTxtConfig;
import net.yacy.utils.CryptoLib;
import net.yacy.utils.UPnP;
import net.yacy.utils.crypt;
import com.google.common.io.Files;
refactoring (usage of constant names for attributes of authentication check)
2014-01-05 04:23:44 +01:00
moved startup execAPIActions call after Jetty startup execAPIActions require http to be up. The 10s sleep was sufficient to allow Jetty to start, but it's more robust to place the call after http is assigned to switchboard/serverSwitch.
2014-01-01 10:28:49 +01:00
import net.yacy.http.YaCyHttpServer;
fixed shutdown authenticaton problem
2014-01-06 01:48:54 +01:00
import net.yacy.http.YaCyLegacyCredential;
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
public final class Switchboard extends serverSwitch {
introduced a second core named 'webgraph'. This core will hold the link structure, but is not filled yet. To have the opportunity of a second core, multi-core functionality had to be implemented to the deep-embedded solr: - migrated the solr_40 directory content to a subdirectory 'collection1'; the previously used default core is now called collection1 - added solr_40/webgraph subdirectory as second core - added a servlet configuration for the second core 'webgraph' in /IndexSchema_p.html - added instance handling as addition to solr connections: all solr connectors are now instances of an solr 'instance' object; this required a complete re-design of the solr embedding - migrated also caching and sharding ontop of new instance handling - migrated the search apis to handle now the access to a specific core, the default core named 'collection1' - migrated the remote solr search interface to access shards of cores; for the yacy remote search the default core is now called 'solr'; using the peer address as solr address - migrated the solr backup and restore process: old backups cannot be used after this migration! - redesign of solr instance handling in all methods which access the instances: they cannot hold copies of these instances any more; the must retrieve the actuall connection object every time they want to write to it (this solves also some bugs when switching the index/network) - added another schema 'solr.webgraph.schema', the old solr.keys.list is replaced by solr.collection.schema
2013-02-21 13:23:55 +01:00
final static String SOLR_COLLECTION_CONFIGURATION_NAME_OLD = "solr.keys.default.list";
add: reset Solr schema filed selection to default button in IndexSchema_p
2013-03-17 03:46:29 +01:00
public final static String SOLR_COLLECTION_CONFIGURATION_NAME = "solr.collection.schema";
public final static String SOLR_WEBGRAPH_CONFIGURATION_NAME = "solr.webgraph.schema";
introduced a second core named 'webgraph'. This core will hold the link structure, but is not filled yet. To have the opportunity of a second core, multi-core functionality had to be implemented to the deep-embedded solr: - migrated the solr_40 directory content to a subdirectory 'collection1'; the previously used default core is now called collection1 - added solr_40/webgraph subdirectory as second core - added a servlet configuration for the second core 'webgraph' in /IndexSchema_p.html - added instance handling as addition to solr connections: all solr connectors are now instances of an solr 'instance' object; this required a complete re-design of the solr embedding - migrated also caching and sharding ontop of new instance handling - migrated the search apis to handle now the access to a specific core, the default core named 'collection1' - migrated the remote solr search interface to access shards of cores; for the yacy remote search the default core is now called 'solr'; using the peer address as solr address - migrated the solr backup and restore process: old backups cannot be used after this migration! - redesign of solr instance handling in all methods which access the instances: they cannot hold copies of these instances any more; the must retrieve the actuall connection object every time they want to write to it (this solves also some bugs when switching the index/network) - added another schema 'solr.webgraph.schema', the old solr.keys.list is replaced by solr.collection.schema
2013-02-21 13:23:55 +01:00
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
// load slots
public static int xstackCrawlSlots = 2000;
public static long lastPPMUpdate = System.currentTimeMillis() - 30000;
private static final int dhtMaxContainerCount = 500;
private int dhtMaxReferenceCount = 1000;
// colored list management
public static SortedSet<String> badwords = new TreeSet<String>(NaturalOrder.naturalComparator);
public static SortedSet<String> stopwords = new TreeSet<String>(NaturalOrder.naturalComparator);
public static SortedSet<String> blueList = null;
- fix stopword handling for RWI see example http://bugs.yacy.net/view.php?id=247 - append language setting specific stopword list - remove unused OVERHANG stack type
2013-06-06 22:07:54 +02:00
// public static HandleSet badwordHashes = null; // not used 2013-06-06
// public static HandleSet blueListHashes = null; // not used 2013-06-06
public static SortedSet<byte[]> stopwordHashes = null;
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
public static Blacklist urlBlacklist = null;
public static WikiParser wikiParser = null;
// storage management
public File htCachePath;
public final File dictionariesPath;
public File listsPath;
public File htDocsPath;
public File workPath;
public File releasePath;
public File networkRoot;
public File queuesRoot;
public File surrogatesInPath;
public File surrogatesOutPath;
public Segment index;
public LoaderDispatcher loader;
public CrawlSwitchboard crawler;
public CrawlQueues crawlQueues;
public CrawlStacker crawlStacker;
public MessageBoard messageDB;
public WikiBoard wikiDB;
public BlogBoard blogDB;
public BlogBoardComments blogCommentDB;
public RobotsTxt robots;
public Map<String, Object[]> outgoingCookies, incomingCookies;
automatically running optimize on solr if user/search is idle for some time
2013-07-14 10:02:08 +02:00
public volatile long proxyLastAccess, localSearchLastAccess, remoteSearchLastAccess, adminAuthenticationLastAccess, optimizeLastRun;
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
public Network yc;
public ResourceObserver observer;
public UserDB userDB;
public BookmarksDB bookmarksDB;
public WebStructureGraph webStructure;
public ConcurrentHashMap<String, TreeSet<Long>> localSearchTracker, remoteSearchTracker; // mappings from requesting host to a TreeSet of Long(access time)
public long indexedPages = 0;
public int searchQueriesRobinsonFromLocal = 0; // absolute counter of all local queries submitted on this peer from a local or autheticated used
public int searchQueriesRobinsonFromRemote = 0; // absolute counter of all local queries submitted on this peer from a remote IP without authentication
public float searchQueriesGlobal = 0f; // partial counter of remote queries (1/number-of-requested-peers)
public SortedMap<byte[], String> clusterhashes; // map of peerhash(String)/alternative-local-address as ip:port or only ip (String) or null if address in seed should be used
public List<Pattern> networkWhitelist, networkBlacklist;
public FilterEngine domainList;
private Dispatcher dhtDispatcher;
public LinkedBlockingQueue<String> trail;
public SeedDB peers;
public WorkTables tables;
public Tray tray;
public WorkflowProcessor<IndexingQueueEntry> indexingDocumentProcessor;
public WorkflowProcessor<IndexingQueueEntry> indexingCondensementProcessor;
public WorkflowProcessor<IndexingQueueEntry> indexingAnalysisProcessor;
public WorkflowProcessor<IndexingQueueEntry> indexingStorageProcessor;
public RobotsTxtConfig robotstxtConfig = null;
public boolean useTailCache;
public boolean exceed134217727;
private final Semaphore shutdownSync = new Semaphore(0);
private boolean terminate = false;
private boolean startupAction = true; // this is set to false after the first event
private static Switchboard sb;
public HashMap<String, Object[]> crawlJobsStatus = new HashMap<String, Object[]>();
- generalized SchemaConfiguration into super-class Configuration and adopted other classes which used the configuration-only access for that class - removed many warnings - adjusted logging
2013-02-25 00:09:41 +01:00
public Switchboard(final File dataPath, final File appPath, final String initPath, final String configPath) {
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
super(dataPath, appPath, initPath, configPath);
sb = this;
// check if port is already occupied
final int port = getConfigInt("port", 8090);
try {
if ( TimeoutRequest.ping(Domains.LOCALHOST, port, 500) ) {
throw new RuntimeException(
"a server is already running on the YaCy port "
+ port
+ "; possibly another YaCy process has not terminated yet. Please stop YaCy before running a new instance.");
}
Added 'final' for all exception blocks as this helps the Java compiler to optimize memory usage Conflicts: source/net/yacy/search/Switchboard.java
2013-07-17 18:31:30 +02:00
} catch (final ExecutionException e1 ) {
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
}
MemoryTracker.startSystemProfiling();
// set loglevel and log
- refactoring of log to ConcurrentLog: jdk-based logger tend to block at java.util.logging.Logger.log(Logger.java:476) in concurrent environments. This makes logging a main performance issue. To overcome this problem, this is a add-on to jdk logging to put log entries on a concurrent message queue and log the messages one by one using a separate process. - FTPClient uses the concurrent logging instead of the log4j logger
2013-07-09 14:28:25 +02:00
setLog(new ConcurrentLog("SWITCHBOARD"));
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
AccessTracker.setDumpFile(new File("DATA/LOG/queries.log"));
// set default peer name
Seed.ANON_PREFIX = getConfig("peernameprefix", "_anon");
// UPnP port mapping
if ( getConfigBool(SwitchboardConstants.UPNP_ENABLED, false) ) {
InstantBusyThread.oneTimeJob(UPnP.class, "addPortMapping", 0);
}
// init TrayIcon if possible
this.tray = new Tray(this);
// remote proxy configuration
initRemoteProxy();
// memory configuration
long tableCachingLimit = getConfigLong("tableCachingLimit", 419430400L);
if ( MemoryControl.available() > tableCachingLimit ) {
this.useTailCache = true;
}
this.exceed134217727 = getConfigBool("exceed134217727", true);
if ( MemoryControl.available() > 1024L * 1024L * 1024L * 2L ) {
this.exceed134217727 = true;
}
// load values from configs
final File indexPath =
getDataPath(SwitchboardConstants.INDEX_PRIMARY_PATH, SwitchboardConstants.INDEX_PATH_DEFAULT);
- refactoring of log to ConcurrentLog: jdk-based logger tend to block at java.util.logging.Logger.log(Logger.java:476) in concurrent environments. This makes logging a main performance issue. To overcome this problem, this is a add-on to jdk logging to put log entries on a concurrent message queue and log the messages one by one using a separate process. - FTPClient uses the concurrent logging instead of the log4j logger
2013-07-09 14:28:25 +02:00
this.log.config("Index Primary Path: " + indexPath.toString());
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
this.listsPath =
getDataPath(SwitchboardConstants.LISTS_PATH, SwitchboardConstants.LISTS_PATH_DEFAULT);
- refactoring of log to ConcurrentLog: jdk-based logger tend to block at java.util.logging.Logger.log(Logger.java:476) in concurrent environments. This makes logging a main performance issue. To overcome this problem, this is a add-on to jdk logging to put log entries on a concurrent message queue and log the messages one by one using a separate process. - FTPClient uses the concurrent logging instead of the log4j logger
2013-07-09 14:28:25 +02:00
this.log.config("Lists Path: " + this.listsPath.toString());
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
this.htDocsPath =
getDataPath(SwitchboardConstants.HTDOCS_PATH, SwitchboardConstants.HTDOCS_PATH_DEFAULT);
- refactoring of log to ConcurrentLog: jdk-based logger tend to block at java.util.logging.Logger.log(Logger.java:476) in concurrent environments. This makes logging a main performance issue. To overcome this problem, this is a add-on to jdk logging to put log entries on a concurrent message queue and log the messages one by one using a separate process. - FTPClient uses the concurrent logging instead of the log4j logger
2013-07-09 14:28:25 +02:00
this.log.config("HTDOCS Path: " + this.htDocsPath.toString());
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
this.workPath = getDataPath(SwitchboardConstants.WORK_PATH, SwitchboardConstants.WORK_PATH_DEFAULT);
this.workPath.mkdirs();
// if default work files exist, copy them (don't overwrite existing!)
File defaultWorkPath = new File("defaults/data/work");
if (defaultWorkPath.list() != null) {
for (String fs : defaultWorkPath.list()) {
File wf = new File(this.workPath, fs);
if (!wf.exists()) {
- generalized SchemaConfiguration into super-class Configuration and adopted other classes which used the configuration-only access for that class - removed many warnings - adjusted logging
2013-02-25 00:09:41 +01:00
try {
Files.copy(new File(defaultWorkPath, fs), wf);
Added 'final' for all exception blocks as this helps the Java compiler to optimize memory usage Conflicts: source/net/yacy/search/Switchboard.java
2013-07-17 18:31:30 +02:00
} catch (final IOException e) {
- refactoring of log to ConcurrentLog: jdk-based logger tend to block at java.util.logging.Logger.log(Logger.java:476) in concurrent environments. This makes logging a main performance issue. To overcome this problem, this is a add-on to jdk logging to put log entries on a concurrent message queue and log the messages one by one using a separate process. - FTPClient uses the concurrent logging instead of the log4j logger
2013-07-09 14:28:25 +02:00
ConcurrentLog.logException(e);
- generalized SchemaConfiguration into super-class Configuration and adopted other classes which used the configuration-only access for that class - removed many warnings - adjusted logging
2013-02-25 00:09:41 +01:00
}
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
}
}
}
- refactoring of log to ConcurrentLog: jdk-based logger tend to block at java.util.logging.Logger.log(Logger.java:476) in concurrent environments. This makes logging a main performance issue. To overcome this problem, this is a add-on to jdk logging to put log entries on a concurrent message queue and log the messages one by one using a separate process. - FTPClient uses the concurrent logging instead of the log4j logger
2013-07-09 14:28:25 +02:00
this.log.config("Work Path: " + this.workPath.toString());
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
this.dictionariesPath =
getDataPath(
SwitchboardConstants.DICTIONARY_SOURCE_PATH,
SwitchboardConstants.DICTIONARY_SOURCE_PATH_DEFAULT);
- refactoring of log to ConcurrentLog: jdk-based logger tend to block at java.util.logging.Logger.log(Logger.java:476) in concurrent environments. This makes logging a main performance issue. To overcome this problem, this is a add-on to jdk logging to put log entries on a concurrent message queue and log the messages one by one using a separate process. - FTPClient uses the concurrent logging instead of the log4j logger
2013-07-09 14:28:25 +02:00
this.log.config("Dictionaries Path:" + this.dictionariesPath.toString());
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
// init libraries
- refactoring of log to ConcurrentLog: jdk-based logger tend to block at java.util.logging.Logger.log(Logger.java:476) in concurrent environments. This makes logging a main performance issue. To overcome this problem, this is a add-on to jdk logging to put log entries on a concurrent message queue and log the messages one by one using a separate process. - FTPClient uses the concurrent logging instead of the log4j logger
2013-07-09 14:28:25 +02:00
this.log.config("initializing libraries");
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
new Thread() {
@Override
public void run() {
Thread.currentThread().setName("LibraryProvider.initialize");
LibraryProvider.initialize(Switchboard.this.dictionariesPath);
}
}.start();
// init global host name cache
Domains.init(new File(this.workPath, "globalhosts.list"));
// init sessionid name file
final String sessionidNamesFile = getConfig("sessionidNamesFile", "defaults/sessionid.names");
- refactoring of log to ConcurrentLog: jdk-based logger tend to block at java.util.logging.Logger.log(Logger.java:476) in concurrent environments. This makes logging a main performance issue. To overcome this problem, this is a add-on to jdk logging to put log entries on a concurrent message queue and log the messages one by one using a separate process. - FTPClient uses the concurrent logging instead of the log4j logger
2013-07-09 14:28:25 +02:00
this.log.config("Loading sessionid file " + sessionidNamesFile);
- the webgraph shall store all links which appear on a web page and not all unique links! This made it necessary, that a large portion of the parser and link processing classes must be adopted to carry a different type of link collection which carry a property attribute which are attached to web anchors. - introduction of a new URL class, AnchorURL - the other url classes, DigestURI and MultiProtocolURI had been renamed and refactored to fit into a new document package schema, document.id - cleanup of net.yacy.cora.document package and refactoring
2013-09-15 00:30:23 +02:00
MultiProtocolURL.initSessionIDNames(FileUtils.loadList(new File(getAppPath(), sessionidNamesFile)));
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
// init tables
this.tables = new WorkTables(this.workPath);
// set a high maximum cache size to current size; this is adopted later automatically
final int wordCacheMaxCount = (int) getConfigLong(SwitchboardConstants.WORDCACHE_MAX_COUNT, 20000);
setConfig(SwitchboardConstants.WORDCACHE_MAX_COUNT, Integer.toString(wordCacheMaxCount));
// load the network definition
- generalized SchemaConfiguration into super-class Configuration and adopted other classes which used the configuration-only access for that class - removed many warnings - adjusted logging
2013-02-25 00:09:41 +01:00
try {
overwriteNetworkDefinition();
Added 'final' for all exception blocks as this helps the Java compiler to optimize memory usage Conflicts: source/net/yacy/search/Switchboard.java
2013-07-17 18:31:30 +02:00
} catch (final FileNotFoundException e) {
- refactoring of log to ConcurrentLog: jdk-based logger tend to block at java.util.logging.Logger.log(Logger.java:476) in concurrent environments. This makes logging a main performance issue. To overcome this problem, this is a add-on to jdk logging to put log entries on a concurrent message queue and log the messages one by one using a separate process. - FTPClient uses the concurrent logging instead of the log4j logger
2013-07-09 14:28:25 +02:00
ConcurrentLog.logException(e);
Added 'final' for all exception blocks as this helps the Java compiler to optimize memory usage Conflicts: source/net/yacy/search/Switchboard.java
2013-07-17 18:31:30 +02:00
} catch (final IOException e) {
- refactoring of log to ConcurrentLog: jdk-based logger tend to block at java.util.logging.Logger.log(Logger.java:476) in concurrent environments. This makes logging a main performance issue. To overcome this problem, this is a add-on to jdk logging to put log entries on a concurrent message queue and log the messages one by one using a separate process. - FTPClient uses the concurrent logging instead of the log4j logger
2013-07-09 14:28:25 +02:00
ConcurrentLog.logException(e);
- generalized SchemaConfiguration into super-class Configuration and adopted other classes which used the configuration-only access for that class - removed many warnings - adjusted logging
2013-02-25 00:09:41 +01:00
}
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
// start indexing management
- refactoring of log to ConcurrentLog: jdk-based logger tend to block at java.util.logging.Logger.log(Logger.java:476) in concurrent environments. This makes logging a main performance issue. To overcome this problem, this is a add-on to jdk logging to put log entries on a concurrent message queue and log the messages one by one using a separate process. - FTPClient uses the concurrent logging instead of the log4j logger
2013-07-09 14:28:25 +02:00
this.log.config("Starting Indexing Management");
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
final String networkName = getConfig(SwitchboardConstants.NETWORK_NAME, "");
final long fileSizeMax = (OS.isWindows) ? this.getConfigLong("filesize.max.win", Integer.MAX_VALUE) : this.getConfigLong( "filesize.max.other", Integer.MAX_VALUE);
final int redundancy = (int) this.getConfigLong("network.unit.dhtredundancy.senior", 1);
final int partitionExponent = (int) this.getConfigLong("network.unit.dht.partitionExponent", 0);
this.networkRoot = new File(new File(indexPath, networkName), "NETWORK");
this.queuesRoot = new File(new File(indexPath, networkName), "QUEUES");
this.networkRoot.mkdirs();
this.queuesRoot.mkdirs();
// prepare a solr index profile switch list
introduced a second core named 'webgraph'. This core will hold the link structure, but is not filled yet. To have the opportunity of a second core, multi-core functionality had to be implemented to the deep-embedded solr: - migrated the solr_40 directory content to a subdirectory 'collection1'; the previously used default core is now called collection1 - added solr_40/webgraph subdirectory as second core - added a servlet configuration for the second core 'webgraph' in /IndexSchema_p.html - added instance handling as addition to solr connections: all solr connectors are now instances of an solr 'instance' object; this required a complete re-design of the solr embedding - migrated also caching and sharding ontop of new instance handling - migrated the search apis to handle now the access to a specific core, the default core named 'collection1' - migrated the remote solr search interface to access shards of cores; for the yacy remote search the default core is now called 'solr'; using the peer address as solr address - migrated the solr backup and restore process: old backups cannot be used after this migration! - redesign of solr instance handling in all methods which access the instances: they cannot hold copies of these instances any more; the must retrieve the actuall connection object every time they want to write to it (this solves also some bugs when switching the index/network) - added another schema 'solr.webgraph.schema', the old solr.keys.list is replaced by solr.collection.schema
2013-02-21 13:23:55 +01:00
final File solrCollectionConfigurationInitFile = new File(getAppPath(), "defaults/" + SOLR_COLLECTION_CONFIGURATION_NAME);
final File solrCollectionConfigurationWorkFile = new File(getDataPath(), "DATA/SETTINGS/" + SOLR_COLLECTION_CONFIGURATION_NAME);
final File solrWebgraphConfigurationInitFile = new File(getAppPath(), "defaults/" + SOLR_WEBGRAPH_CONFIGURATION_NAME);
final File solrWebgraphConfigurationWorkFile = new File(getDataPath(), "DATA/SETTINGS/" + SOLR_WEBGRAPH_CONFIGURATION_NAME);
- generalized SchemaConfiguration into super-class Configuration and adopted other classes which used the configuration-only access for that class - removed many warnings - adjusted logging
2013-02-25 00:09:41 +01:00
CollectionConfiguration solrCollectionConfigurationWork = null;
WebgraphConfiguration solrWebgraphConfigurationWork = null;
changes in ranking computation - an existing ranking servlet for solr was extended. It is now possible to set boost values for fields, boost functions and boost queries. - The ranking can have different instances, but currently only the first one is used - added an abstraction layer for fields which can be used for search and those fields can be edited in the solr ranking configruation - the ranking value from solr within the field score is used to combine remote search requests, which all are created using the same locally defined boost values - reduced the number of fields which are used for search (makes it faster) - replaced some text fields by string fields (makes indexing faster) - removed classes which had no use - made a large number of experiments for a better ranking and created a temporary setting which prefers hits inside titles - adjusted also the RWI-based ranking computation to 'prefer title' - made special cases like for portal search where no post-processing and post-ranking is wanted: this keeps the original ranking order as done by Solr - fixed many bugs with old settings for ranking
2013-03-13 14:47:00 +01:00
introduced a second core named 'webgraph'. This core will hold the link structure, but is not filled yet. To have the opportunity of a second core, multi-core functionality had to be implemented to the deep-embedded solr: - migrated the solr_40 directory content to a subdirectory 'collection1'; the previously used default core is now called collection1 - added solr_40/webgraph subdirectory as second core - added a servlet configuration for the second core 'webgraph' in /IndexSchema_p.html - added instance handling as addition to solr connections: all solr connectors are now instances of an solr 'instance' object; this required a complete re-design of the solr embedding - migrated also caching and sharding ontop of new instance handling - migrated the search apis to handle now the access to a specific core, the default core named 'collection1' - migrated the remote solr search interface to access shards of cores; for the yacy remote search the default core is now called 'solr'; using the peer address as solr address - migrated the solr backup and restore process: old backups cannot be used after this migration! - redesign of solr instance handling in all methods which access the instances: they cannot hold copies of these instances any more; the must retrieve the actuall connection object every time they want to write to it (this solves also some bugs when switching the index/network) - added another schema 'solr.webgraph.schema', the old solr.keys.list is replaced by solr.collection.schema
2013-02-21 13:23:55 +01:00
// migrate the old Schema file path to a new one
final File solrCollectionConfigurationWorkOldFile = new File(getDataPath(), "DATA/SETTINGS/" + SOLR_COLLECTION_CONFIGURATION_NAME_OLD);
if (solrCollectionConfigurationWorkOldFile.exists() && !solrCollectionConfigurationWorkFile.exists()) solrCollectionConfigurationWorkOldFile.renameTo(solrCollectionConfigurationWorkFile);
changes in ranking computation - an existing ranking servlet for solr was extended. It is now possible to set boost values for fields, boost functions and boost queries. - The ranking can have different instances, but currently only the first one is used - added an abstraction layer for fields which can be used for search and those fields can be edited in the solr ranking configruation - the ranking value from solr within the field score is used to combine remote search requests, which all are created using the same locally defined boost values - reduced the number of fields which are used for search (makes it faster) - replaced some text fields by string fields (makes indexing faster) - removed classes which had no use - made a large number of experiments for a better ranking and created a temporary setting which prefers hits inside titles - adjusted also the RWI-based ranking computation to 'prefer title' - made special cases like for portal search where no post-processing and post-ranking is wanted: this keeps the original ranking order as done by Solr - fixed many bugs with old settings for ranking
2013-03-13 14:47:00 +01:00
- generalized SchemaConfiguration into super-class Configuration and adopted other classes which used the configuration-only access for that class - removed many warnings - adjusted logging
2013-02-25 00:09:41 +01:00
// initialize the collection schema if it does not yet exist
if (!solrCollectionConfigurationWorkFile.exists()) try {
Files.copy(solrCollectionConfigurationInitFile, solrCollectionConfigurationWorkFile);
Added 'final' for all exception blocks as this helps the Java compiler to optimize memory usage Conflicts: source/net/yacy/search/Switchboard.java
2013-07-17 18:31:30 +02:00
} catch (final IOException e) {ConcurrentLog.logException(e);}
changes in ranking computation - an existing ranking servlet for solr was extended. It is now possible to set boost values for fields, boost functions and boost queries. - The ranking can have different instances, but currently only the first one is used - added an abstraction layer for fields which can be used for search and those fields can be edited in the solr ranking configruation - the ranking value from solr within the field score is used to combine remote search requests, which all are created using the same locally defined boost values - reduced the number of fields which are used for search (makes it faster) - replaced some text fields by string fields (makes indexing faster) - removed classes which had no use - made a large number of experiments for a better ranking and created a temporary setting which prefers hits inside titles - adjusted also the RWI-based ranking computation to 'prefer title' - made special cases like for portal search where no post-processing and post-ranking is wanted: this keeps the original ranking order as done by Solr - fixed many bugs with old settings for ranking
2013-03-13 14:47:00 +01:00
introduced a second core named 'webgraph'. This core will hold the link structure, but is not filled yet. To have the opportunity of a second core, multi-core functionality had to be implemented to the deep-embedded solr: - migrated the solr_40 directory content to a subdirectory 'collection1'; the previously used default core is now called collection1 - added solr_40/webgraph subdirectory as second core - added a servlet configuration for the second core 'webgraph' in /IndexSchema_p.html - added instance handling as addition to solr connections: all solr connectors are now instances of an solr 'instance' object; this required a complete re-design of the solr embedding - migrated also caching and sharding ontop of new instance handling - migrated the search apis to handle now the access to a specific core, the default core named 'collection1' - migrated the remote solr search interface to access shards of cores; for the yacy remote search the default core is now called 'solr'; using the peer address as solr address - migrated the solr backup and restore process: old backups cannot be used after this migration! - redesign of solr instance handling in all methods which access the instances: they cannot hold copies of these instances any more; the must retrieve the actuall connection object every time they want to write to it (this solves also some bugs when switching the index/network) - added another schema 'solr.webgraph.schema', the old solr.keys.list is replaced by solr.collection.schema
2013-02-21 13:23:55 +01:00
// lazy definition of schema: do not write empty fields
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
final boolean solrlazy = getConfigBool(SwitchboardConstants.FEDERATED_SERVICE_SOLR_INDEXING_LAZY, true);
changes in ranking computation - an existing ranking servlet for solr was extended. It is now possible to set boost values for fields, boost functions and boost queries. - The ranking can have different instances, but currently only the first one is used - added an abstraction layer for fields which can be used for search and those fields can be edited in the solr ranking configruation - the ranking value from solr within the field score is used to combine remote search requests, which all are created using the same locally defined boost values - reduced the number of fields which are used for search (makes it faster) - replaced some text fields by string fields (makes indexing faster) - removed classes which had no use - made a large number of experiments for a better ranking and created a temporary setting which prefers hits inside titles - adjusted also the RWI-based ranking computation to 'prefer title' - made special cases like for portal search where no post-processing and post-ranking is wanted: this keeps the original ranking order as done by Solr - fixed many bugs with old settings for ranking
2013-03-13 14:47:00 +01:00
introduced a second core named 'webgraph'. This core will hold the link structure, but is not filled yet. To have the opportunity of a second core, multi-core functionality had to be implemented to the deep-embedded solr: - migrated the solr_40 directory content to a subdirectory 'collection1'; the previously used default core is now called collection1 - added solr_40/webgraph subdirectory as second core - added a servlet configuration for the second core 'webgraph' in /IndexSchema_p.html - added instance handling as addition to solr connections: all solr connectors are now instances of an solr 'instance' object; this required a complete re-design of the solr embedding - migrated also caching and sharding ontop of new instance handling - migrated the search apis to handle now the access to a specific core, the default core named 'collection1' - migrated the remote solr search interface to access shards of cores; for the yacy remote search the default core is now called 'solr'; using the peer address as solr address - migrated the solr backup and restore process: old backups cannot be used after this migration! - redesign of solr instance handling in all methods which access the instances: they cannot hold copies of these instances any more; the must retrieve the actuall connection object every time they want to write to it (this solves also some bugs when switching the index/network) - added another schema 'solr.webgraph.schema', the old solr.keys.list is replaced by solr.collection.schema
2013-02-21 13:23:55 +01:00
// define collection schema
- generalized SchemaConfiguration into super-class Configuration and adopted other classes which used the configuration-only access for that class - removed many warnings - adjusted logging
2013-02-25 00:09:41 +01:00
try {
final CollectionConfiguration solrCollectionConfigurationInit = new CollectionConfiguration(solrCollectionConfigurationInitFile, solrlazy);
solrCollectionConfigurationWork = new CollectionConfiguration(solrCollectionConfigurationWorkFile, solrlazy);
// update the working scheme with the backup scheme. This is necessary to include new features.
// new features are always activated by default (if activated in input-backupScheme)
solrCollectionConfigurationWork.fill(solrCollectionConfigurationInit, true);
// switch on some fields which are necessary for ranking and faceting
for (CollectionSchema field: new CollectionSchema[]{
CollectionSchema.host_s, CollectionSchema.load_date_dt,
CollectionSchema.url_file_ext_s, CollectionSchema.last_modified, // needed for media search and /date operator
/*YaCySchema.url_paths_sxt,*/ CollectionSchema.host_organization_s, // needed to search in the url
link strings should not be tokenized
2013-09-01 14:35:36 +02:00
/*YaCySchema.inboundlinks_protocol_sxt,*/ CollectionSchema.inboundlinks_urlstub_sxt, // needed for HostBrowser
/*YaCySchema.outboundlinks_protocol_sxt,*/ CollectionSchema.outboundlinks_urlstub_sxt,// needed to enhance the crawler
added httpstatus_i to automatically switched on fields (used in all search queries)
2013-04-27 03:11:44 +02:00
CollectionSchema.httpstatus_i // used in all search queries to filter out error documents
- generalized SchemaConfiguration into super-class Configuration and adopted other classes which used the configuration-only access for that class - removed many warnings - adjusted logging
2013-02-25 00:09:41 +01:00
}) {
SchemaConfiguration.Entry entry = solrCollectionConfigurationWork.get(field.name()); entry.setEnable(true); solrCollectionConfigurationWork.put(field.name(), entry);
}
now writing image alt texts and (camelcase-)parsed urls into a text search field for a better image retrieval
2013-06-18 16:51:56 +02:00
// activate some fields that are necessary here
solrCollectionConfigurationWork.get(CollectionSchema.images_urlstub_sxt.getSolrFieldName()).setEnable(true);
- generalized SchemaConfiguration into super-class Configuration and adopted other classes which used the configuration-only access for that class - removed many warnings - adjusted logging
2013-02-25 00:09:41 +01:00
solrCollectionConfigurationWork.commit();
Added 'final' for all exception blocks as this helps the Java compiler to optimize memory usage Conflicts: source/net/yacy/search/Switchboard.java
2013-07-17 18:31:30 +02:00
} catch (final IOException e) {ConcurrentLog.logException(e);}
introduced a second core named 'webgraph'. This core will hold the link structure, but is not filled yet. To have the opportunity of a second core, multi-core functionality had to be implemented to the deep-embedded solr: - migrated the solr_40 directory content to a subdirectory 'collection1'; the previously used default core is now called collection1 - added solr_40/webgraph subdirectory as second core - added a servlet configuration for the second core 'webgraph' in /IndexSchema_p.html - added instance handling as addition to solr connections: all solr connectors are now instances of an solr 'instance' object; this required a complete re-design of the solr embedding - migrated also caching and sharding ontop of new instance handling - migrated the search apis to handle now the access to a specific core, the default core named 'collection1' - migrated the remote solr search interface to access shards of cores; for the yacy remote search the default core is now called 'solr'; using the peer address as solr address - migrated the solr backup and restore process: old backups cannot be used after this migration! - redesign of solr instance handling in all methods which access the instances: they cannot hold copies of these instances any more; the must retrieve the actuall connection object every time they want to write to it (this solves also some bugs when switching the index/network) - added another schema 'solr.webgraph.schema', the old solr.keys.list is replaced by solr.collection.schema
2013-02-21 13:23:55 +01:00
- generalized SchemaConfiguration into super-class Configuration and adopted other classes which used the configuration-only access for that class - removed many warnings - adjusted logging
2013-02-25 00:09:41 +01:00
// initialize the webgraph schema if it does not yet exist
if (!solrWebgraphConfigurationWorkFile.exists()) try {
Files.copy(solrWebgraphConfigurationInitFile, solrWebgraphConfigurationWorkFile);
Added 'final' for all exception blocks as this helps the Java compiler to optimize memory usage Conflicts: source/net/yacy/search/Switchboard.java
2013-07-17 18:31:30 +02:00
} catch (final IOException e) {ConcurrentLog.logException(e);}
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
- generalized SchemaConfiguration into super-class Configuration and adopted other classes which used the configuration-only access for that class - removed many warnings - adjusted logging
2013-02-25 00:09:41 +01:00
// define webgraph schema
try {
final WebgraphConfiguration solrWebgraphConfigurationInit = new WebgraphConfiguration(solrWebgraphConfigurationInitFile, solrlazy);
solrWebgraphConfigurationWork = new WebgraphConfiguration(solrWebgraphConfigurationWorkFile, solrlazy);
solrWebgraphConfigurationWork.fill(solrWebgraphConfigurationInit, true);
solrWebgraphConfigurationWork.commit();
Added 'final' for all exception blocks as this helps the Java compiler to optimize memory usage Conflicts: source/net/yacy/search/Switchboard.java
2013-07-17 18:31:30 +02:00
} catch (final IOException e) {ConcurrentLog.logException(e);}
- generalized SchemaConfiguration into super-class Configuration and adopted other classes which used the configuration-only access for that class - removed many warnings - adjusted logging
2013-02-25 00:09:41 +01:00
changes in ranking computation - an existing ranking servlet for solr was extended. It is now possible to set boost values for fields, boost functions and boost queries. - The ranking can have different instances, but currently only the first one is used - added an abstraction layer for fields which can be used for search and those fields can be edited in the solr ranking configruation - the ranking value from solr within the field score is used to combine remote search requests, which all are created using the same locally defined boost values - reduced the number of fields which are used for search (makes it faster) - replaced some text fields by string fields (makes indexing faster) - removed classes which had no use - made a large number of experiments for a better ranking and created a temporary setting which prefers hits inside titles - adjusted also the RWI-based ranking computation to 'prefer title' - made special cases like for portal search where no post-processing and post-ranking is wanted: this keeps the original ranking order as done by Solr - fixed many bugs with old settings for ranking
2013-03-13 14:47:00 +01:00
// define boosts
Ranking.setMinTokenLen(this.getConfigInt(SwitchboardConstants.SEARCH_RANKING_SOLR_DOUBLEDETECTION_MINLENGTH, 3));
Ranking.setQuantRate(this.getConfigFloat(SwitchboardConstants.SEARCH_RANKING_SOLR_DOUBLEDETECTION_QUANTRATE, 0.5f));
for (int i = 0; i <= 3; i++) {
// must be done every time the boosts change
Ranking r = solrCollectionConfigurationWork.getRanking(i);
enhanced ranking; patches to replace old ranking
2013-10-09 15:10:03 +02:00
String name = this.getConfig(SwitchboardConstants.SEARCH_RANKING_SOLR_COLLECTION_BOOSTNAME_ + i, "_dummy" + i);
String boosts = this.getConfig(SwitchboardConstants.SEARCH_RANKING_SOLR_COLLECTION_BOOSTFIELDS_ + i, "text_t^1.0");
String bq = this.getConfig(SwitchboardConstants.SEARCH_RANKING_SOLR_COLLECTION_BOOSTQUERY_ + i, "");
String bf = this.getConfig(SwitchboardConstants.SEARCH_RANKING_SOLR_COLLECTION_BOOSTFUNCTION_ + i, "");
// apply some hard-coded patches for earlier experiments we do not want any more
if (bf.equals("product(recip(rord(last_modified),1,1000,1000),div(product(log(product(references_external_i,references_exthosts_i)),div(references_internal_i,host_extent_i)),add(clickdepth_i,1)))") ||
bf.equals("scale(cr_host_norm_i,1,20)")) bf = "";
if (i == 0 && bq.equals("fuzzy_signature_unique_b:true^100000.0")) bq = "clickdepth_i:0^0.8 clickdepth_i:1^0.4";
if (boosts.equals("url_paths_sxt^1000.0,synonyms_sxt^1.0,title^10000.0,text_t^2.0,h1_txt^1000.0,h2_txt^100.0,host_organization_s^100000.0")) boosts = "url_paths_sxt^3.0,synonyms_sxt^0.5,title^5.0,text_t^1.0,host_s^6.0,h1_txt^5.0,url_file_name_tokens_t^4.0,h2_txt^2.0";
r.setName(name);
r.updateBoosts(boosts);
r.setBoostQuery(bq);
r.setBoostFunction(bf);
changes in ranking computation - an existing ranking servlet for solr was extended. It is now possible to set boost values for fields, boost functions and boost queries. - The ranking can have different instances, but currently only the first one is used - added an abstraction layer for fields which can be used for search and those fields can be edited in the solr ranking configruation - the ranking value from solr within the field score is used to combine remote search requests, which all are created using the same locally defined boost values - reduced the number of fields which are used for search (makes it faster) - replaced some text fields by string fields (makes indexing faster) - removed classes which had no use - made a large number of experiments for a better ranking and created a temporary setting which prefers hits inside titles - adjusted also the RWI-based ranking computation to 'prefer title' - made special cases like for portal search where no post-processing and post-ranking is wanted: this keeps the original ranking order as done by Solr - fixed many bugs with old settings for ranking
2013-03-13 14:47:00 +01:00
}
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
// initialize index
ReferenceContainer.maxReferences = getConfigInt("index.maxReferences", 0);
final File segmentsPath = new File(new File(indexPath, networkName), "SEGMENTS");
introduced a second core named 'webgraph'. This core will hold the link structure, but is not filled yet. To have the opportunity of a second core, multi-core functionality had to be implemented to the deep-embedded solr: - migrated the solr_40 directory content to a subdirectory 'collection1'; the previously used default core is now called collection1 - added solr_40/webgraph subdirectory as second core - added a servlet configuration for the second core 'webgraph' in /IndexSchema_p.html - added instance handling as addition to solr connections: all solr connectors are now instances of an solr 'instance' object; this required a complete re-design of the solr embedding - migrated also caching and sharding ontop of new instance handling - migrated the search apis to handle now the access to a specific core, the default core named 'collection1' - migrated the remote solr search interface to access shards of cores; for the yacy remote search the default core is now called 'solr'; using the peer address as solr address - migrated the solr backup and restore process: old backups cannot be used after this migration! - redesign of solr instance handling in all methods which access the instances: they cannot hold copies of these instances any more; the must retrieve the actuall connection object every time they want to write to it (this solves also some bugs when switching the index/network) - added another schema 'solr.webgraph.schema', the old solr.keys.list is replaced by solr.collection.schema
2013-02-21 13:23:55 +01:00
this.index = new Segment(this.log, segmentsPath, solrCollectionConfigurationWork, solrWebgraphConfigurationWork);
- generalized SchemaConfiguration into super-class Configuration and adopted other classes which used the configuration-only access for that class - removed many warnings - adjusted logging
2013-02-25 00:09:41 +01:00
if (this.getConfigBool(SwitchboardConstants.CORE_SERVICE_RWI, true)) try {
this.index.connectRWI(wordCacheMaxCount, fileSizeMax);
Added 'final' for all exception blocks as this helps the Java compiler to optimize memory usage Conflicts: source/net/yacy/search/Switchboard.java
2013-07-17 18:31:30 +02:00
} catch (final IOException e) {ConcurrentLog.logException(e);}
- generalized SchemaConfiguration into super-class Configuration and adopted other classes which used the configuration-only access for that class - removed many warnings - adjusted logging
2013-02-25 00:09:41 +01:00
if (this.getConfigBool(SwitchboardConstants.CORE_SERVICE_CITATION, true)) try {
this.index.connectCitation(wordCacheMaxCount, fileSizeMax);
Added 'final' for all exception blocks as this helps the Java compiler to optimize memory usage Conflicts: source/net/yacy/search/Switchboard.java
2013-07-17 18:31:30 +02:00
} catch (final IOException e) {ConcurrentLog.logException(e);}
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
if (this.getConfigBool(SwitchboardConstants.CORE_SERVICE_FULLTEXT, true)) {
this.index.connectUrlDb(this.useTailCache, this.exceed134217727);
Added 'final' for all exception blocks as this helps the Java compiler to optimize memory usage Conflicts: source/net/yacy/search/Switchboard.java
2013-07-17 18:31:30 +02:00
try {this.index.fulltext().connectLocalSolr();} catch (final IOException e) {ConcurrentLog.logException(e);}
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
}
enhanced webgraph processing
2013-12-04 01:54:45 +01:00
this.index.fulltext().setUseWebgraph(this.getConfigBool(SwitchboardConstants.CORE_SERVICE_WEBGRAPH, false));
changes in ranking computation - an existing ranking servlet for solr was extended. It is now possible to set boost values for fields, boost functions and boost queries. - The ranking can have different instances, but currently only the first one is used - added an abstraction layer for fields which can be used for search and those fields can be edited in the solr ranking configruation - the ranking value from solr within the field score is used to combine remote search requests, which all are created using the same locally defined boost values - reduced the number of fields which are used for search (makes it faster) - replaced some text fields by string fields (makes indexing faster) - removed classes which had no use - made a large number of experiments for a better ranking and created a temporary setting which prefers hits inside titles - adjusted also the RWI-based ranking computation to 'prefer title' - made special cases like for portal search where no post-processing and post-ranking is wanted: this keeps the original ranking order as done by Solr - fixed many bugs with old settings for ranking
2013-03-13 14:47:00 +01:00
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
// set up the solr interface
final String solrurls = getConfig(SwitchboardConstants.FEDERATED_SERVICE_SOLR_INDEXING_URL, "http://127.0.0.1:8983/solr");
final boolean usesolr = getConfigBool(SwitchboardConstants.FEDERATED_SERVICE_SOLR_INDEXING_ENABLED, false) & solrurls.length() > 0;
tried to fix timeout and connection-lost problems when using an outside solr.
2013-11-28 01:31:53 +01:00
final int solrtimeout = getConfigInt(SwitchboardConstants.FEDERATED_SERVICE_SOLR_INDEXING_TIMEOUT, 60000);
added option to attach remote solr servers in read-only mode
2013-12-27 02:55:21 +01:00
final boolean writeEnabled = getConfigBool(SwitchboardConstants.FEDERATED_SERVICE_SOLR_INDEXING_WRITEENABLED, true);
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
if (usesolr && solrurls != null && solrurls.length() > 0) {
try {
added the property federated.service.solr.indexing.timeout to yacy.init to provide a configurable time-out for solr; see also: http://bugs.yacy.net/view.php?id=254
2013-07-22 17:45:12 +02:00
ArrayList<RemoteInstance> instances = RemoteInstance.getShardInstances(solrurls, null, null, solrtimeout);
added option to attach remote solr servers in read-only mode
2013-12-27 02:55:21 +01:00
this.index.fulltext().connectRemoteSolr(instances, writeEnabled);
Added 'final' for all exception blocks as this helps the Java compiler to optimize memory usage Conflicts: source/net/yacy/search/Switchboard.java
2013-07-17 18:31:30 +02:00
} catch (final IOException e ) {
- refactoring of log to ConcurrentLog: jdk-based logger tend to block at java.util.logging.Logger.log(Logger.java:476) in concurrent environments. This makes logging a main performance issue. To overcome this problem, this is a add-on to jdk logging to put log entries on a concurrent message queue and log the messages one by one using a separate process. - FTPClient uses the concurrent logging instead of the log4j logger
2013-07-09 14:28:25 +02:00
ConcurrentLog.logException(e);
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
}
}
re-declared some fields to be of type string rather than text which makes them more efficient and less large
2013-05-06 16:45:54 +02:00
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
// initialize network database
final File mySeedFile = new File(this.networkRoot, SeedDB.DBFILE_OWN_SEED);
this.peers =
new SeedDB(
this.networkRoot,
"seed.new.heap",
"seed.old.heap",
"seed.pot.heap",
mySeedFile,
redundancy,
partitionExponent,
false,
this.exceed134217727);
// load domainList
try {
this.domainList = null;
if ( !getConfig("network.unit.domainlist", "").equals("") ) {
final Reader r =
getConfigFileFromWebOrLocally(getConfig("network.unit.domainlist", ""), getAppPath()
.getAbsolutePath(), new File(this.networkRoot, "domainlist.txt"));
this.domainList = new FilterEngine();
this.domainList.loadList(new BufferedReader(r), null);
}
Added 'final' for all exception blocks as this helps the Java compiler to optimize memory usage Conflicts: source/net/yacy/search/Switchboard.java
2013-07-17 18:31:30 +02:00
} catch (final FileNotFoundException e ) {
- refactoring of log to ConcurrentLog: jdk-based logger tend to block at java.util.logging.Logger.log(Logger.java:476) in concurrent environments. This makes logging a main performance issue. To overcome this problem, this is a add-on to jdk logging to put log entries on a concurrent message queue and log the messages one by one using a separate process. - FTPClient uses the concurrent logging instead of the log4j logger
2013-07-09 14:28:25 +02:00
this.log.severe("CONFIG: domainlist not found: " + e.getMessage());
Added 'final' for all exception blocks as this helps the Java compiler to optimize memory usage Conflicts: source/net/yacy/search/Switchboard.java
2013-07-17 18:31:30 +02:00
} catch (final IOException e ) {
- refactoring of log to ConcurrentLog: jdk-based logger tend to block at java.util.logging.Logger.log(Logger.java:476) in concurrent environments. This makes logging a main performance issue. To overcome this problem, this is a add-on to jdk logging to put log entries on a concurrent message queue and log the messages one by one using a separate process. - FTPClient uses the concurrent logging instead of the log4j logger
2013-07-09 14:28:25 +02:00
this.log.severe("CONFIG: error while retrieving domainlist: " + e.getMessage());
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
}
// create a crawler
- crawl profiles shall be deleted from active and passive stacks if they are deleted to terminate the crawl because otherwise the crawl will go on after the load-from-passive stack policy. - better check if a crawl is terminated using the loader queue.
2013-09-26 10:22:31 +02:00
this.crawler = new CrawlSwitchboard(networkName, this);
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
// start yacy core
- refactoring of log to ConcurrentLog: jdk-based logger tend to block at java.util.logging.Logger.log(Logger.java:476) in concurrent environments. This makes logging a main performance issue. To overcome this problem, this is a add-on to jdk logging to put log entries on a concurrent message queue and log the messages one by one using a separate process. - FTPClient uses the concurrent logging instead of the log4j logger
2013-07-09 14:28:25 +02:00
this.log.config("Starting YaCy Protocol Core");
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
this.yc = new Network(this);
InstantBusyThread.oneTimeJob(this, "loadSeedLists", 0);
//final long startedSeedListAquisition = System.currentTimeMillis();
// init a DHT transmission dispatcher
this.dhtDispatcher =
(this.peers.sizeConnected() == 0) ? null : new Dispatcher(
this.index,
this.peers,
true,
10000);
// set up local robots.txt
this.robotstxtConfig = RobotsTxtConfig.init(this);
// setting timestamp of last proxy access
this.proxyLastAccess = System.currentTimeMillis() - 10000;
this.localSearchLastAccess = System.currentTimeMillis() - 10000;
this.remoteSearchLastAccess = System.currentTimeMillis() - 10000;
automatically running optimize on solr if user/search is idle for some time
2013-07-14 10:02:08 +02:00
this.adminAuthenticationLastAccess = System.currentTimeMillis();
this.optimizeLastRun = System.currentTimeMillis();
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
this.webStructure = new WebStructureGraph(new File(this.queuesRoot, "webStructure.map"));
// configuring list path
if ( !(this.listsPath.exists()) ) {
this.listsPath.mkdirs();
}
// load coloured lists
if ( blueList == null ) {
// read only once upon first instantiation of this class
final String f =
getConfig(SwitchboardConstants.LIST_BLUE, SwitchboardConstants.LIST_BLUE_DEFAULT);
final File plasmaBlueListFile = new File(f);
if ( f != null ) {
blueList = SetTools.loadList(plasmaBlueListFile, NaturalOrder.naturalComparator);
} else {
blueList = new TreeSet<String>();
}
- fix stopword handling for RWI see example http://bugs.yacy.net/view.php?id=247 - append language setting specific stopword list - remove unused OVERHANG stack type
2013-06-06 22:07:54 +02:00
// blueListHashes = Word.words2hashesHandles(blueList);
- refactoring of log to ConcurrentLog: jdk-based logger tend to block at java.util.logging.Logger.log(Logger.java:476) in concurrent environments. This makes logging a main performance issue. To overcome this problem, this is a add-on to jdk logging to put log entries on a concurrent message queue and log the messages one by one using a separate process. - FTPClient uses the concurrent logging instead of the log4j logger
2013-07-09 14:28:25 +02:00
this.log.config("loaded blue-list from file "
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
+ plasmaBlueListFile.getName()
+ ", "
+ blueList.size()
+ " entries, "
+ ppRamString(plasmaBlueListFile.length() / 1024));
}
// load blacklist
- refactoring of log to ConcurrentLog: jdk-based logger tend to block at java.util.logging.Logger.log(Logger.java:476) in concurrent environments. This makes logging a main performance issue. To overcome this problem, this is a add-on to jdk logging to put log entries on a concurrent message queue and log the messages one by one using a separate process. - FTPClient uses the concurrent logging instead of the log4j logger
2013-07-09 14:28:25 +02:00
this.log.config("Loading blacklist ...");
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
final File blacklistsPath =
getDataPath(SwitchboardConstants.LISTS_PATH, SwitchboardConstants.LISTS_PATH_DEFAULT);
urlBlacklist = new Blacklist(blacklistsPath);
ListManager.switchboard = this;
ListManager.listsPath = blacklistsPath;
ListManager.reloadBlacklists();
// load badwords (to filter the topwords)
if ( badwords == null || badwords.isEmpty() ) {
final File badwordsFile = new File(appPath, SwitchboardConstants.LIST_BADWORDS_DEFAULT);
badwords = SetTools.loadList(badwordsFile, NaturalOrder.naturalComparator);
- fix stopword handling for RWI see example http://bugs.yacy.net/view.php?id=247 - append language setting specific stopword list - remove unused OVERHANG stack type
2013-06-06 22:07:54 +02:00
// badwordHashes = Word.words2hashesHandles(badwords);
- refactoring of log to ConcurrentLog: jdk-based logger tend to block at java.util.logging.Logger.log(Logger.java:476) in concurrent environments. This makes logging a main performance issue. To overcome this problem, this is a add-on to jdk logging to put log entries on a concurrent message queue and log the messages one by one using a separate process. - FTPClient uses the concurrent logging instead of the log4j logger
2013-07-09 14:28:25 +02:00
this.log.config("loaded badwords from file "
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
+ badwordsFile.getName()
+ ", "
+ badwords.size()
+ " entries, "
+ ppRamString(badwordsFile.length() / 1024));
}
// load stopwords
if ( stopwords == null || stopwords.isEmpty() ) {
final File stopwordsFile = new File(appPath, SwitchboardConstants.LIST_STOPWORDS_DEFAULT);
stopwords = SetTools.loadList(stopwordsFile, NaturalOrder.naturalComparator);
- fix stopword handling for RWI see example http://bugs.yacy.net/view.php?id=247 - append language setting specific stopword list - remove unused OVERHANG stack type
2013-06-06 22:07:54 +02:00
// append locale language stopwords using setting of interface language (file yacy.stopwords.xx)
//TODO: append / share Solr stopwords.txt
final File stopwordsFilelocale = new File (stopwordsFile.getAbsolutePath()+"."+this.getConfig("locale.language","default"));
if (stopwordsFilelocale.exists()) {
stopwords.addAll(SetTools.loadList(stopwordsFilelocale, NaturalOrder.naturalComparator));
}
if (!stopwords.isEmpty()) {
stopwordHashes = new TreeSet<byte[]>(NaturalOrder.naturalOrder);
for (final String wordstr : stopwords) {
stopwordHashes.add(Word.word2hash(wordstr));
}
}
- refactoring of log to ConcurrentLog: jdk-based logger tend to block at java.util.logging.Logger.log(Logger.java:476) in concurrent environments. This makes logging a main performance issue. To overcome this problem, this is a add-on to jdk logging to put log entries on a concurrent message queue and log the messages one by one using a separate process. - FTPClient uses the concurrent logging instead of the log4j logger
2013-07-09 14:28:25 +02:00
this.log.config("loaded stopwords from file "
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
+ stopwordsFile.getName()
+ ", "
+ stopwords.size()
+ " entries, "
+ ppRamString(stopwordsFile.length() / 1024));
}
// start a cache manager
- refactoring of log to ConcurrentLog: jdk-based logger tend to block at java.util.logging.Logger.log(Logger.java:476) in concurrent environments. This makes logging a main performance issue. To overcome this problem, this is a add-on to jdk logging to put log entries on a concurrent message queue and log the messages one by one using a separate process. - FTPClient uses the concurrent logging instead of the log4j logger
2013-07-09 14:28:25 +02:00
this.log.config("Starting HT Cache Manager");
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
// create the cache directory
this.htCachePath =
getDataPath(SwitchboardConstants.HTCACHE_PATH, SwitchboardConstants.HTCACHE_PATH_DEFAULT);
- refactoring of log to ConcurrentLog: jdk-based logger tend to block at java.util.logging.Logger.log(Logger.java:476) in concurrent environments. This makes logging a main performance issue. To overcome this problem, this is a add-on to jdk logging to put log entries on a concurrent message queue and log the messages one by one using a separate process. - FTPClient uses the concurrent logging instead of the log4j logger
2013-07-09 14:28:25 +02:00
this.log.info("HTCACHE Path = " + this.htCachePath.getAbsolutePath());
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
final long maxCacheSize =
1024L * 1024L * Long.parseLong(getConfig(SwitchboardConstants.PROXY_CACHE_SIZE, "2")); // this is megabyte
Cache.init(this.htCachePath, this.peers.mySeed().hash, maxCacheSize);
// create the surrogates directories
this.surrogatesInPath =
getDataPath(
SwitchboardConstants.SURROGATES_IN_PATH,
SwitchboardConstants.SURROGATES_IN_PATH_DEFAULT);
- refactoring of log to ConcurrentLog: jdk-based logger tend to block at java.util.logging.Logger.log(Logger.java:476) in concurrent environments. This makes logging a main performance issue. To overcome this problem, this is a add-on to jdk logging to put log entries on a concurrent message queue and log the messages one by one using a separate process. - FTPClient uses the concurrent logging instead of the log4j logger
2013-07-09 14:28:25 +02:00
this.log.info("surrogates.in Path = " + this.surrogatesInPath.getAbsolutePath());
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
this.surrogatesInPath.mkdirs();
this.surrogatesOutPath =
getDataPath(
SwitchboardConstants.SURROGATES_OUT_PATH,
SwitchboardConstants.SURROGATES_OUT_PATH_DEFAULT);
- refactoring of log to ConcurrentLog: jdk-based logger tend to block at java.util.logging.Logger.log(Logger.java:476) in concurrent environments. This makes logging a main performance issue. To overcome this problem, this is a add-on to jdk logging to put log entries on a concurrent message queue and log the messages one by one using a separate process. - FTPClient uses the concurrent logging instead of the log4j logger
2013-07-09 14:28:25 +02:00
this.log.info("surrogates.out Path = " + this.surrogatesOutPath.getAbsolutePath());
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
this.surrogatesOutPath.mkdirs();
// create the release download directory
this.releasePath =
getDataPath(SwitchboardConstants.RELEASE_PATH, SwitchboardConstants.RELEASE_PATH_DEFAULT);
this.releasePath.mkdirs();
- refactoring of log to ConcurrentLog: jdk-based logger tend to block at java.util.logging.Logger.log(Logger.java:476) in concurrent environments. This makes logging a main performance issue. To overcome this problem, this is a add-on to jdk logging to put log entries on a concurrent message queue and log the messages one by one using a separate process. - FTPClient uses the concurrent logging instead of the log4j logger
2013-07-09 14:28:25 +02:00
this.log.info("RELEASE Path = " + this.releasePath.getAbsolutePath());
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
// starting message board
- generalized SchemaConfiguration into super-class Configuration and adopted other classes which used the configuration-only access for that class - removed many warnings - adjusted logging
2013-02-25 00:09:41 +01:00
try {
initMessages();
Added 'final' for all exception blocks as this helps the Java compiler to optimize memory usage Conflicts: source/net/yacy/search/Switchboard.java
2013-07-17 18:31:30 +02:00
} catch (final IOException e) {
- refactoring of log to ConcurrentLog: jdk-based logger tend to block at java.util.logging.Logger.log(Logger.java:476) in concurrent environments. This makes logging a main performance issue. To overcome this problem, this is a add-on to jdk logging to put log entries on a concurrent message queue and log the messages one by one using a separate process. - FTPClient uses the concurrent logging instead of the log4j logger
2013-07-09 14:28:25 +02:00
ConcurrentLog.logException(e);
- generalized SchemaConfiguration into super-class Configuration and adopted other classes which used the configuration-only access for that class - removed many warnings - adjusted logging
2013-02-25 00:09:41 +01:00
}
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
// starting wiki
- generalized SchemaConfiguration into super-class Configuration and adopted other classes which used the configuration-only access for that class - removed many warnings - adjusted logging
2013-02-25 00:09:41 +01:00
try {
initWiki();
Added 'final' for all exception blocks as this helps the Java compiler to optimize memory usage Conflicts: source/net/yacy/search/Switchboard.java
2013-07-17 18:31:30 +02:00
} catch (final IOException e) {
- refactoring of log to ConcurrentLog: jdk-based logger tend to block at java.util.logging.Logger.log(Logger.java:476) in concurrent environments. This makes logging a main performance issue. To overcome this problem, this is a add-on to jdk logging to put log entries on a concurrent message queue and log the messages one by one using a separate process. - FTPClient uses the concurrent logging instead of the log4j logger
2013-07-09 14:28:25 +02:00
ConcurrentLog.logException(e);
- generalized SchemaConfiguration into super-class Configuration and adopted other classes which used the configuration-only access for that class - removed many warnings - adjusted logging
2013-02-25 00:09:41 +01:00
}
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
//starting blog
- generalized SchemaConfiguration into super-class Configuration and adopted other classes which used the configuration-only access for that class - removed many warnings - adjusted logging
2013-02-25 00:09:41 +01:00
try {
initBlog();
Added 'final' for all exception blocks as this helps the Java compiler to optimize memory usage Conflicts: source/net/yacy/search/Switchboard.java
2013-07-17 18:31:30 +02:00
} catch (final IOException e) {
- refactoring of log to ConcurrentLog: jdk-based logger tend to block at java.util.logging.Logger.log(Logger.java:476) in concurrent environments. This makes logging a main performance issue. To overcome this problem, this is a add-on to jdk logging to put log entries on a concurrent message queue and log the messages one by one using a separate process. - FTPClient uses the concurrent logging instead of the log4j logger
2013-07-09 14:28:25 +02:00
ConcurrentLog.logException(e);
- generalized SchemaConfiguration into super-class Configuration and adopted other classes which used the configuration-only access for that class - removed many warnings - adjusted logging
2013-02-25 00:09:41 +01:00
}
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
// init User DB
- refactoring of log to ConcurrentLog: jdk-based logger tend to block at java.util.logging.Logger.log(Logger.java:476) in concurrent environments. This makes logging a main performance issue. To overcome this problem, this is a add-on to jdk logging to put log entries on a concurrent message queue and log the messages one by one using a separate process. - FTPClient uses the concurrent logging instead of the log4j logger
2013-07-09 14:28:25 +02:00
this.log.config("Loading User DB");
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
final File userDbFile = new File(getDataPath(), "DATA/SETTINGS/user.heap");
- generalized SchemaConfiguration into super-class Configuration and adopted other classes which used the configuration-only access for that class - removed many warnings - adjusted logging
2013-02-25 00:09:41 +01:00
try {
this.userDB = new UserDB(userDbFile);
- refactoring of log to ConcurrentLog: jdk-based logger tend to block at java.util.logging.Logger.log(Logger.java:476) in concurrent environments. This makes logging a main performance issue. To overcome this problem, this is a add-on to jdk logging to put log entries on a concurrent message queue and log the messages one by one using a separate process. - FTPClient uses the concurrent logging instead of the log4j logger
2013-07-09 14:28:25 +02:00
this.log.config("Loaded User DB from file "
- generalized SchemaConfiguration into super-class Configuration and adopted other classes which used the configuration-only access for that class - removed many warnings - adjusted logging
2013-02-25 00:09:41 +01:00
+ userDbFile.getName()
+ ", "
+ this.userDB.size()
+ " entries"
+ ", "
+ ppRamString(userDbFile.length() / 1024));
Added 'final' for all exception blocks as this helps the Java compiler to optimize memory usage Conflicts: source/net/yacy/search/Switchboard.java
2013-07-17 18:31:30 +02:00
} catch (final IOException e) {
- refactoring of log to ConcurrentLog: jdk-based logger tend to block at java.util.logging.Logger.log(Logger.java:476) in concurrent environments. This makes logging a main performance issue. To overcome this problem, this is a add-on to jdk logging to put log entries on a concurrent message queue and log the messages one by one using a separate process. - FTPClient uses the concurrent logging instead of the log4j logger
2013-07-09 14:28:25 +02:00
ConcurrentLog.logException(e);
- generalized SchemaConfiguration into super-class Configuration and adopted other classes which used the configuration-only access for that class - removed many warnings - adjusted logging
2013-02-25 00:09:41 +01:00
}
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
- generalized SchemaConfiguration into super-class Configuration and adopted other classes which used the configuration-only access for that class - removed many warnings - adjusted logging
2013-02-25 00:09:41 +01:00
// init user triplestores
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
JenaTripleStore.initPrivateStores();
// init html parser evaluation scheme
File parserPropertiesPath = new File("defaults/");
String[] settingsList = parserPropertiesPath.list();
for ( final String l : settingsList ) {
if ( l.startsWith("parser.") && l.endsWith(".properties") ) {
- generalized SchemaConfiguration into super-class Configuration and adopted other classes which used the configuration-only access for that class - removed many warnings - adjusted logging
2013-02-25 00:09:41 +01:00
try {
Evaluation.add(new File(parserPropertiesPath, l));
Added 'final' for all exception blocks as this helps the Java compiler to optimize memory usage Conflicts: source/net/yacy/search/Switchboard.java
2013-07-17 18:31:30 +02:00
} catch (final IOException e) {
- refactoring of log to ConcurrentLog: jdk-based logger tend to block at java.util.logging.Logger.log(Logger.java:476) in concurrent environments. This makes logging a main performance issue. To overcome this problem, this is a add-on to jdk logging to put log entries on a concurrent message queue and log the messages one by one using a separate process. - FTPClient uses the concurrent logging instead of the log4j logger
2013-07-09 14:28:25 +02:00
ConcurrentLog.logException(e);
- generalized SchemaConfiguration into super-class Configuration and adopted other classes which used the configuration-only access for that class - removed many warnings - adjusted logging
2013-02-25 00:09:41 +01:00
}
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
}
}
parserPropertiesPath = new File(getDataPath(), "DATA/SETTINGS/");
settingsList = parserPropertiesPath.list();
for ( final String l : settingsList ) {
if ( l.startsWith("parser.") && l.endsWith(".properties") ) {
- generalized SchemaConfiguration into super-class Configuration and adopted other classes which used the configuration-only access for that class - removed many warnings - adjusted logging
2013-02-25 00:09:41 +01:00
try {
Evaluation.add(new File(parserPropertiesPath, l));
Added 'final' for all exception blocks as this helps the Java compiler to optimize memory usage Conflicts: source/net/yacy/search/Switchboard.java
2013-07-17 18:31:30 +02:00
} catch (final IOException e) {
- refactoring of log to ConcurrentLog: jdk-based logger tend to block at java.util.logging.Logger.log(Logger.java:476) in concurrent environments. This makes logging a main performance issue. To overcome this problem, this is a add-on to jdk logging to put log entries on a concurrent message queue and log the messages one by one using a separate process. - FTPClient uses the concurrent logging instead of the log4j logger
2013-07-09 14:28:25 +02:00
ConcurrentLog.logException(e);
- generalized SchemaConfiguration into super-class Configuration and adopted other classes which used the configuration-only access for that class - removed many warnings - adjusted logging
2013-02-25 00:09:41 +01:00
}
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
}
}
// init bookmarks DB: needs more time since this does a DNS lookup for each Bookmark.
// Can be started concurrently
new Thread() {
@Override
public void run() {
Thread.currentThread().setName("Switchboard.initBookmarks");
try {
initBookmarks();
Added 'final' for all exception blocks as this helps the Java compiler to optimize memory usage Conflicts: source/net/yacy/search/Switchboard.java
2013-07-17 18:31:30 +02:00
} catch (final IOException e ) {
- refactoring of log to ConcurrentLog: jdk-based logger tend to block at java.util.logging.Logger.log(Logger.java:476) in concurrent environments. This makes logging a main performance issue. To overcome this problem, this is a add-on to jdk logging to put log entries on a concurrent message queue and log the messages one by one using a separate process. - FTPClient uses the concurrent logging instead of the log4j logger
2013-07-09 14:28:25 +02:00
ConcurrentLog.logException(e);
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
}
}
}.start();
// define a realtime parsable mimetype list
- refactoring of log to ConcurrentLog: jdk-based logger tend to block at java.util.logging.Logger.log(Logger.java:476) in concurrent environments. This makes logging a main performance issue. To overcome this problem, this is a add-on to jdk logging to put log entries on a concurrent message queue and log the messages one by one using a separate process. - FTPClient uses the concurrent logging instead of the log4j logger
2013-07-09 14:28:25 +02:00
this.log.config("Parser: Initializing Mime Type deny list");
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
final boolean enableAudioTags = getConfigBool("parser.enableAudioTags", false);
- refactoring of log to ConcurrentLog: jdk-based logger tend to block at java.util.logging.Logger.log(Logger.java:476) in concurrent environments. This makes logging a main performance issue. To overcome this problem, this is a add-on to jdk logging to put log entries on a concurrent message queue and log the messages one by one using a separate process. - FTPClient uses the concurrent logging instead of the log4j logger
2013-07-09 14:28:25 +02:00
log.config("Parser: parser.enableAudioTags= "+enableAudioTags);
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
final StringBuilder denyExt = new StringBuilder(256);
final StringBuilder denyMime = new StringBuilder(256);
denyExt.append(getConfig(SwitchboardConstants.PARSER_MIME_DENY, ""));
denyMime.append(getConfig(SwitchboardConstants.PARSER_EXTENSIONS_DENY, ""));
if (!enableAudioTags) {
if(denyExt.length()>0) {
denyExt.append(audioTagParser.SEPERATOR);
}
denyExt.append(audioTagParser.EXTENSIONS);
if(denyMime.length()>0) {
denyMime.append(audioTagParser.SEPERATOR);
}
denyMime.append(audioTagParser.MIME_TYPES);
setConfig(SwitchboardConstants.PARSER_EXTENSIONS_DENY, denyExt.toString());
setConfig(SwitchboardConstants.PARSER_MIME_DENY, denyMime.toString());
setConfig("parser.enableAudioTags", true);
}
TextParser.setDenyMime(getConfig(SwitchboardConstants.PARSER_MIME_DENY, ""));
TextParser.setDenyExtension(getConfig(SwitchboardConstants.PARSER_EXTENSIONS_DENY, ""));
// start a loader
- refactoring of log to ConcurrentLog: jdk-based logger tend to block at java.util.logging.Logger.log(Logger.java:476) in concurrent environments. This makes logging a main performance issue. To overcome this problem, this is a add-on to jdk logging to put log entries on a concurrent message queue and log the messages one by one using a separate process. - FTPClient uses the concurrent logging instead of the log4j logger
2013-07-09 14:28:25 +02:00
this.log.config("Starting Crawl Loader");
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
this.loader = new LoaderDispatcher(this);
// load the robots.txt db
- refactoring of log to ConcurrentLog: jdk-based logger tend to block at java.util.logging.Logger.log(Logger.java:476) in concurrent environments. This makes logging a main performance issue. To overcome this problem, this is a add-on to jdk logging to put log entries on a concurrent message queue and log the messages one by one using a separate process. - FTPClient uses the concurrent logging instead of the log4j logger
2013-07-09 14:28:25 +02:00
this.log.config("Initializing robots.txt DB");
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
this.robots = new RobotsTxt(this.tables, this.loader);
- generalized SchemaConfiguration into super-class Configuration and adopted other classes which used the configuration-only access for that class - removed many warnings - adjusted logging
2013-02-25 00:09:41 +01:00
try {
- refactoring of log to ConcurrentLog: jdk-based logger tend to block at java.util.logging.Logger.log(Logger.java:476) in concurrent environments. This makes logging a main performance issue. To overcome this problem, this is a add-on to jdk logging to put log entries on a concurrent message queue and log the messages one by one using a separate process. - FTPClient uses the concurrent logging instead of the log4j logger
2013-07-09 14:28:25 +02:00
this.log.config("Loaded robots.txt DB: " + this.robots.size() + " entries");
Added 'final' for all exception blocks as this helps the Java compiler to optimize memory usage Conflicts: source/net/yacy/search/Switchboard.java
2013-07-17 18:31:30 +02:00
} catch (final IOException e) {
- refactoring of log to ConcurrentLog: jdk-based logger tend to block at java.util.logging.Logger.log(Logger.java:476) in concurrent environments. This makes logging a main performance issue. To overcome this problem, this is a add-on to jdk logging to put log entries on a concurrent message queue and log the messages one by one using a separate process. - FTPClient uses the concurrent logging instead of the log4j logger
2013-07-09 14:28:25 +02:00
ConcurrentLog.logException(e);
- generalized SchemaConfiguration into super-class Configuration and adopted other classes which used the configuration-only access for that class - removed many warnings - adjusted logging
2013-02-25 00:09:41 +01:00
}
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
// load oai tables
final Map<String, File> oaiFriends =
OAIListFriendsLoader.loadListFriendsSources(
new File("defaults/oaiListFriendsSource.xml"),
getDataPath());
Redesign of crawler identification and robots steering. A non-p2p user in intranets and the internet can now choose to appear as Googlebot. This is an essential necessity to be able to compete in the field of commercial search appliances, since most web pages are these days optimized only for Google and no other search platform any more. All commercial search engine providers have a built-in fake-Google User Agent to be able to get the same search index as Google can do. Without the resistance against obeying to robots.txt in this case, no competition is possible any more. YaCy will always obey the robots.txt when it is used for crawling the web in a peer-to-peer network, but to establish a Search Appliance (like a Google Search Appliance, GSA) it is necessary to be able to behave exactly like a Google crawler. With this change, you will be able to switch the user agent when portal or intranet mode is selected on per-crawl-start basis. Every crawl start can have a different user agent.
2013-08-22 14:23:47 +02:00
OAIListFriendsLoader.init(this.loader, oaiFriends, ClientIdentification.yacyInternetCrawlerAgent);
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
this.crawlQueues = new CrawlQueues(this, this.queuesRoot);
resume paused crawls on startup; user expects that restarts 'heal' everything
2013-04-11 15:07:08 +02:00
// on startup, resume all crawls
setConfig(SwitchboardConstants.CRAWLJOB_LOCAL_CRAWL + "_isPaused", "false");
setConfig(SwitchboardConstants.CRAWLJOB_LOCAL_CRAWL + "_isPaused_cause", "");
setConfig(SwitchboardConstants.CRAWLJOB_REMOTE_TRIGGERED_CRAWL + "_isPaused", "false");
setConfig(SwitchboardConstants.CRAWLJOB_REMOTE_TRIGGERED_CRAWL + "_isPaused_cause", "");
setConfig(SwitchboardConstants.CRAWLJOB_REMOTE_CRAWL_LOADER + "_isPaused", "false");
setConfig(SwitchboardConstants.CRAWLJOB_REMOTE_CRAWL_LOADER + "_isPaused_cause", "");
this.crawlJobsStatus.put(SwitchboardConstants.CRAWLJOB_LOCAL_CRAWL, new Object[] {new Object(), false});
this.crawlJobsStatus.put(SwitchboardConstants.CRAWLJOB_REMOTE_TRIGGERED_CRAWL, new Object[] {new Object(), false});
this.crawlJobsStatus.put(SwitchboardConstants.CRAWLJOB_REMOTE_CRAWL_LOADER, new Object[] {new Object(), false});
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
// init cookie-Monitor
- refactoring of log to ConcurrentLog: jdk-based logger tend to block at java.util.logging.Logger.log(Logger.java:476) in concurrent environments. This makes logging a main performance issue. To overcome this problem, this is a add-on to jdk logging to put log entries on a concurrent message queue and log the messages one by one using a separate process. - FTPClient uses the concurrent logging instead of the log4j logger
2013-07-09 14:28:25 +02:00
this.log.config("Starting Cookie Monitor");
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
this.outgoingCookies = new ConcurrentHashMap<String, Object[]>();
this.incomingCookies = new ConcurrentHashMap<String, Object[]>();
// init search history trackers
this.localSearchTracker = new ConcurrentHashMap<String, TreeSet<Long>>(); // String:TreeSet - IP:set of Long(accessTime)
this.remoteSearchTracker = new ConcurrentHashMap<String, TreeSet<Long>>();
// init messages: clean up message symbol
final File notifierSource =
new File(getAppPath(), getConfig(
SwitchboardConstants.HTROOT_PATH,
SwitchboardConstants.HTROOT_PATH_DEFAULT) + "/env/grafics/empty.gif");
final File notifierDest =
new File(
getDataPath(SwitchboardConstants.HTDOCS_PATH, SwitchboardConstants.HTDOCS_PATH_DEFAULT),
"notifier.gif");
try {
Files.copy(notifierSource, notifierDest);
Added 'final' for all exception blocks as this helps the Java compiler to optimize memory usage Conflicts: source/net/yacy/search/Switchboard.java
2013-07-17 18:31:30 +02:00
} catch (final IOException e ) {
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
}
// init nameCacheNoCachingList
try {
Domains.setNoCachingPatterns(getConfig(SwitchboardConstants.HTTPC_NAME_CACHE_CACHING_PATTERNS_NO, ""));
Added 'final' for all exception blocks as this helps the Java compiler to optimize memory usage Conflicts: source/net/yacy/search/Switchboard.java
2013-07-17 18:31:30 +02:00
} catch (final PatternSyntaxException pse) {
- refactoring of log to ConcurrentLog: jdk-based logger tend to block at java.util.logging.Logger.log(Logger.java:476) in concurrent environments. This makes logging a main performance issue. To overcome this problem, this is a add-on to jdk logging to put log entries on a concurrent message queue and log the messages one by one using a separate process. - FTPClient uses the concurrent logging instead of the log4j logger
2013-07-09 14:28:25 +02:00
ConcurrentLog.severe("Switchboard", "Invalid regular expression in "
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
+ SwitchboardConstants.HTTPC_NAME_CACHE_CACHING_PATTERNS_NO
+ " property: " + pse.getMessage());
System.exit(-1);
}
// generate snippets cache
- refactoring of log to ConcurrentLog: jdk-based logger tend to block at java.util.logging.Logger.log(Logger.java:476) in concurrent environments. This makes logging a main performance issue. To overcome this problem, this is a add-on to jdk logging to put log entries on a concurrent message queue and log the messages one by one using a separate process. - FTPClient uses the concurrent logging instead of the log4j logger
2013-07-09 14:28:25 +02:00
this.log.config("Initializing Snippet Cache");
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
// init the wiki
wikiParser = new WikiCode();
// initializing the resourceObserver
InstantBusyThread.oneTimeJob(ResourceObserver.class, "initThread", 0);
// initializing the stackCrawlThread
this.crawlStacker =
new CrawlStacker(
this.robots,
this.crawlQueues,
this.crawler,
this.index,
this.peers,
isIntranetMode(),
isGlobalMode(),
this.domainList); // Intranet and Global mode may be both true!
// possibly switch off localIP check
Domains.setNoLocalCheck(isAllIPMode());
// check status of account configuration: when local url crawling is allowed, it is not allowed
// that an automatic authorization of localhost is done, because in this case crawls from local
// addresses are blocked to prevent attack szenarios where remote pages contain links to localhost
// addresses that can steer a YaCy peer
refactoring (usage of constant names for attributes of authentication check)
2014-01-05 04:23:44 +01:00
if ( !getConfigBool(SwitchboardConstants.ADMIN_ACCOUNT_FOR_LOCALHOST, false) ) {
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
if ( getConfig(SwitchboardConstants.ADMIN_ACCOUNT_B64MD5, "").startsWith("0000") ) {
// the password was set automatically with a random value.
// We must remove that here to prevent that a user cannot log in any more
setConfig(SwitchboardConstants.ADMIN_ACCOUNT_B64MD5, "");
// after this a message must be generated to alert the user to set a new password
- refactoring of log to ConcurrentLog: jdk-based logger tend to block at java.util.logging.Logger.log(Logger.java:476) in concurrent environments. This makes logging a main performance issue. To overcome this problem, this is a add-on to jdk logging to put log entries on a concurrent message queue and log the messages one by one using a separate process. - FTPClient uses the concurrent logging instead of the log4j logger
2013-07-09 14:28:25 +02:00
this.log.info("RANDOM PASSWORD REMOVED! User must set a new password");
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
}
}
// initializing dht chunk generation
this.dhtMaxReferenceCount = (int) getConfigLong(SwitchboardConstants.INDEX_DIST_CHUNK_SIZE_START, 50);
// init robinson cluster
// before we do that, we wait some time until the seed list is loaded.
this.clusterhashes = this.peers.clusterHashes(getConfig("cluster.peers.yacydomain", ""));
// deploy blocking threads
this.indexingStorageProcessor =
new WorkflowProcessor<IndexingQueueEntry>(
"storeDocumentIndex",
"This is the sequencing step of the indexing queue. Files are written as streams, too much councurrency would destroy IO performance. In this process the words are written to the RWI cache, which flushes if it is full.",
new String[] {
"RWI/Cache/Collections"
},
this,
"storeDocumentIndex",
2,
null,
- reduction of the concurrently running processes to make YaCy more adjusted to smaller and 1-core devices. - the workflow processor now starts no process at all. these are started as soon as parser/condenser/indexing queues are filled. - better abstraction
2013-04-25 11:33:17 +02:00
1);
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
this.indexingAnalysisProcessor =
new WorkflowProcessor<IndexingQueueEntry>(
"webStructureAnalysis",
"This just stores the link structure of the document into a web structure database.",
new String[] {
"storeDocumentIndex"
},
this,
"webStructureAnalysis",
WorkflowProcessor.availableCPU + 1,
this.indexingStorageProcessor,
WorkflowProcessor.availableCPU);
this.indexingCondensementProcessor =
new WorkflowProcessor<IndexingQueueEntry>(
"condenseDocument",
"This does a structural analysis of plain texts: markup of headlines, slicing into phrases (i.e. sentences), markup with position, counting of words, calculation of term frequency.",
new String[] {
"webStructureAnalysis"
},
this,
"condenseDocument",
WorkflowProcessor.availableCPU + 1,
this.indexingAnalysisProcessor,
WorkflowProcessor.availableCPU);
this.indexingDocumentProcessor =
new WorkflowProcessor<IndexingQueueEntry>(
"parseDocument",
"This does the parsing of the newly loaded documents from the web. The result is not only a plain text document, but also a list of URLs that are embedded into the document. The urls are handed over to the CrawlStacker. This process has two child process queues!",
new String[] {
"condenseDocument", "CrawlStacker"
},
this,
"parseDocument",
Math.max(20, WorkflowProcessor.availableCPU * 2), // it may happen that this is filled with new files from the search process. That means there should be enough place for two result pages
this.indexingCondensementProcessor,
WorkflowProcessor.availableCPU);
// deploy busy threads
- refactoring of log to ConcurrentLog: jdk-based logger tend to block at java.util.logging.Logger.log(Logger.java:476) in concurrent environments. This makes logging a main performance issue. To overcome this problem, this is a add-on to jdk logging to put log entries on a concurrent message queue and log the messages one by one using a separate process. - FTPClient uses the concurrent logging instead of the log4j logger
2013-07-09 14:28:25 +02:00
this.log.config("Starting Threads");
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
MemoryControl.gc(10000, "plasmaSwitchboard, help for profiler"); // help for profiler - thq
deployThread(
SwitchboardConstants.CLEANUP,
"Cleanup",
"simple cleaning process for monitoring information",
null,
new InstantBusyThread(
this,
SwitchboardConstants.CLEANUP_METHOD_START,
SwitchboardConstants.CLEANUP_METHOD_JOBCOUNT,
SwitchboardConstants.CLEANUP_METHOD_FREEMEM,
60000,
Long.MAX_VALUE,
10000,
Long.MAX_VALUE),
60000); // all 5 Minutes, wait 1 minute until first run
deployThread(
SwitchboardConstants.SURROGATES,
"Surrogates",
"A thread that polls the SURROGATES path and puts all Documents in one surroagte file into the indexing queue.",
null,
new InstantBusyThread(
this,
SwitchboardConstants.SURROGATES_METHOD_START,
SwitchboardConstants.SURROGATES_METHOD_JOBCOUNT,
SwitchboardConstants.SURROGATES_METHOD_FREEMEM,
20000,
Long.MAX_VALUE,
0,
Long.MAX_VALUE),
10000);
deployThread(
SwitchboardConstants.CRAWLJOB_REMOTE_TRIGGERED_CRAWL,
"Remote Crawl Job",
"thread that performes a single crawl/indexing step triggered by a remote peer",
- fix monitor url of crawl job in PerformanceQueues_p.html - reduce logging of every index add (switch embeddedsolr.add from info to debug)
2013-05-10 04:38:13 +02:00
"/IndexCreateQueues_p.html?stack=REMOTE",
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
new InstantBusyThread(
this.crawlQueues,
SwitchboardConstants.CRAWLJOB_REMOTE_TRIGGERED_CRAWL_METHOD_START,
SwitchboardConstants.CRAWLJOB_REMOTE_TRIGGERED_CRAWL_METHOD_JOBCOUNT,
SwitchboardConstants.CRAWLJOB_REMOTE_TRIGGERED_CRAWL_METHOD_FREEMEM,
0,
Long.MAX_VALUE,
0,
Long.MAX_VALUE),
10000);
deployThread(
SwitchboardConstants.CRAWLJOB_REMOTE_CRAWL_LOADER,
"Remote Crawl URL Loader",
"thread that loads remote crawl lists from other peers",
null,
new InstantBusyThread(
this.crawlQueues,
SwitchboardConstants.CRAWLJOB_REMOTE_CRAWL_LOADER_METHOD_START,
SwitchboardConstants.CRAWLJOB_REMOTE_CRAWL_LOADER_METHOD_JOBCOUNT,
SwitchboardConstants.CRAWLJOB_REMOTE_CRAWL_LOADER_METHOD_FREEMEM,
10000,
Long.MAX_VALUE,
10000,
Long.MAX_VALUE),
10000); // error here?
deployThread(
SwitchboardConstants.CRAWLJOB_LOCAL_CRAWL,
"Local Crawl",
"thread that performes a single crawl step from the local crawl queue",
- fix monitor url of crawl job in PerformanceQueues_p.html - reduce logging of every index add (switch embeddedsolr.add from info to debug)
2013-05-10 04:38:13 +02:00
"/IndexCreateQueues_p.html?stack=LOCAL",
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
new InstantBusyThread(
this.crawlQueues,
SwitchboardConstants.CRAWLJOB_LOCAL_CRAWL_METHOD_START,
SwitchboardConstants.CRAWLJOB_LOCAL_CRAWL_METHOD_JOBCOUNT,
SwitchboardConstants.CRAWLJOB_LOCAL_CRAWL_METHOD_FREEMEM,
0,
Long.MAX_VALUE,
0,
Long.MAX_VALUE),
10000);
deployThread(
SwitchboardConstants.SEED_UPLOAD,
"Seed-List Upload",
"task that a principal peer performes to generate and upload a seed-list to a ftp account",
null,
new InstantBusyThread(
this.yc,
SwitchboardConstants.SEED_UPLOAD_METHOD_START,
SwitchboardConstants.SEED_UPLOAD_METHOD_JOBCOUNT,
SwitchboardConstants.SEED_UPLOAD_METHOD_FREEMEM,
600000,
Long.MAX_VALUE,
300000,
Long.MAX_VALUE),
180000);
deployThread(
SwitchboardConstants.PEER_PING,
"YaCy Core",
"this is the p2p-control and peer-ping task",
null,
new InstantBusyThread(
this.yc,
SwitchboardConstants.PEER_PING_METHOD_START,
SwitchboardConstants.PEER_PING_METHOD_JOBCOUNT,
SwitchboardConstants.PEER_PING_METHOD_FREEMEM,
30000,
Long.MAX_VALUE,
30000,
Long.MAX_VALUE),
wait a bit longer before doing the first peer ping
2013-07-27 11:00:35 +02:00
10000);
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
deployThread(
SwitchboardConstants.INDEX_DIST,
"DHT Distribution",
"selection, transfer and deletion of index entries that are not searched on your peer, but on others",
null,
new InstantBusyThread(
this,
SwitchboardConstants.INDEX_DIST_METHOD_START,
SwitchboardConstants.INDEX_DIST_METHOD_JOBCOUNT,
SwitchboardConstants.INDEX_DIST_METHOD_FREEMEM,
10000,
Long.MAX_VALUE,
1000,
Long.MAX_VALUE),
delayed first peer-ping to send the first ping out after the http got up; if the ping comes before the http is up, it cannot be recognized as senior peer (if at all). See also: http://bugs.yacy.net/view.php?id=266
2013-07-22 18:21:37 +02:00
60000,
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
Long.parseLong(getConfig(SwitchboardConstants.INDEX_DIST_IDLESLEEP, "5000")),
Long.parseLong(getConfig(SwitchboardConstants.INDEX_DIST_BUSYSLEEP, "0")),
Long.parseLong(getConfig(SwitchboardConstants.INDEX_DIST_MEMPREREQ, "1000000")));
// content control: initialize list sync thread
deployThread(
"720_ccimport",
"Content Control Import",
"this is the content control import thread",
null,
new InstantBusyThread(
new SMWListSyncThread(this, sb.getConfig("contentcontrol.bookmarklist", "contentcontrol"), "Category:Content Source", "/?Url/?Filter/?Category/?Modification date", sb.getConfigBool(
"contentcontrol.smwimport.purgelistoninit", false)),
"run",
SwitchboardConstants.PEER_PING_METHOD_JOBCOUNT,
SwitchboardConstants.PEER_PING_METHOD_FREEMEM,
3000,
10000,
3000,
10000),
2000);
deployThread(
"730_ccfilter",
"Content Control Filter",
"this is the content control filter update thread",
null,
new InstantBusyThread(
new ContentControlFilterUpdateThread(this),
"run",
SwitchboardConstants.PEER_PING_METHOD_JOBCOUNT,
SwitchboardConstants.PEER_PING_METHOD_FREEMEM,
3000,
10000,
3000,
10000),
2000);
// set network-specific performance attributes
if ( this.firstInit ) {
setRemotecrawlPPM(Math.max(1, (int) getConfigLong("network.unit.remotecrawl.speed", 60)));
}
// test routine for snippet fetch
//Set query = new HashSet();
//query.add(CrawlSwitchboardEntry.word2hash("Weitergabe"));
//query.add(CrawlSwitchboardEntry.word2hash("Zahl"));
//plasmaSnippetCache.result scr = snippetCache.retrieve(new URL("http://www.heise.de/mobil/newsticker/meldung/mail/54980"), query, true);
//plasmaSnippetCache.result scr = snippetCache.retrieve(new URL("http://www.heise.de/security/news/foren/go.shtml?read=1&msg_id=7301419&forum_id=72721"), query, true);
//plasmaSnippetCache.result scr = snippetCache.retrieve(new URL("http://www.heise.de/kiosk/archiv/ct/2003/4/20"), query, true, 260);
this.trail = new LinkedBlockingQueue<String>();
moved startup execAPIActions call after Jetty startup execAPIActions require http to be up. The 10s sleep was sufficient to allow Jetty to start, but it's more robust to place the call after http is assigned to switchboard/serverSwitch.
2014-01-01 10:28:49 +01:00
this.log.config("Finished Switchboard Initialization");
}
@Override
public void setHttpServer(YaCyHttpServer server) {
super.setHttpServer(server);
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
// finally start jobs which shall be started after start-up
new Thread() {
public void run() {
moved startup execAPIActions call after Jetty startup execAPIActions require http to be up. The 10s sleep was sufficient to allow Jetty to start, but it's more robust to place the call after http is assigned to switchboard/serverSwitch.
2014-01-01 10:28:49 +01:00
try {Thread.sleep(10000);} catch (final InterruptedException e) {} // needs httpd up
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
execAPIActions(); // trigger startup actions
}
moved startup execAPIActions call after Jetty startup execAPIActions require http to be up. The 10s sleep was sufficient to allow Jetty to start, but it's more robust to place the call after http is assigned to switchboard/serverSwitch.
2014-01-01 10:28:49 +01:00
}.start();
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
}
moved startup execAPIActions call after Jetty startup execAPIActions require http to be up. The 10s sleep was sufficient to allow Jetty to start, but it's more robust to place the call after http is assigned to switchboard/serverSwitch.
2014-01-01 10:28:49 +01:00
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
public int getIndexingProcessorsQueueSize() {
refactoring of WorkflowProcessor, added process counter, update of process counter if an blocking thread dies. Added also a new column in PerformanceConcurrency_p servlet to show the actual number of concurrent processes.
2013-05-13 13:28:07 +02:00
return this.indexingDocumentProcessor.getQueueSize()
+ this.indexingCondensementProcessor.getQueueSize()
+ this.indexingAnalysisProcessor.getQueueSize()
+ this.indexingStorageProcessor.getQueueSize();
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
}
public void overwriteNetworkDefinition() throws FileNotFoundException, IOException {
// load network configuration into settings
String networkUnitDefinition =
getConfig("network.unit.definition", "defaults/yacy.network.freeworld.unit");
if (networkUnitDefinition.isEmpty()) networkUnitDefinition = "defaults/yacy.network.freeworld.unit"; // patch for a strange failure case where the path was overwritten by empty string
// patch old values
if ( networkUnitDefinition.equals("yacy.network.unit") ) {
networkUnitDefinition = "defaults/yacy.network.freeworld.unit";
setConfig("network.unit.definition", networkUnitDefinition);
}
// remove old release and bootstrap locations
final Iterator<String> ki = configKeys();
final ArrayList<String> d = new ArrayList<String>();
String k;
while ( ki.hasNext() ) {
k = ki.next();
if ( k.startsWith("network.unit.update.location") || k.startsWith("network.unit.bootstrap") ) {
d.add(k);
}
}
for ( final String s : d ) {
removeConfig(s); // must be removed afterwards otherwise a ki.remove() would not remove the property on file
}
// include additional network definition properties into our settings
// note that these properties cannot be set in the application because they are
// _always_ overwritten each time with the default values. This is done so on purpose.
// the network definition should be made either consistent for all peers,
// or independently using a bootstrap URL
Map<String, String> initProps;
final Reader netDefReader =
getConfigFileFromWebOrLocally(networkUnitDefinition, getAppPath().getAbsolutePath(), new File(
this.workPath,
"network.definition.backup"));
initProps = FileUtils.table(netDefReader);
setConfig(initProps);
// set release locations
int i = 0;
CryptoLib cryptoLib;
try {
cryptoLib = new CryptoLib();
while ( true ) {
final String location = getConfig("network.unit.update.location" + i, "");
if ( location.isEmpty() ) {
break;
}
- the webgraph shall store all links which appear on a web page and not all unique links! This made it necessary, that a large portion of the parser and link processing classes must be adopted to carry a different type of link collection which carry a property attribute which are attached to web anchors. - introduction of a new URL class, AnchorURL - the other url classes, DigestURI and MultiProtocolURI had been renamed and refactored to fit into a new document package schema, document.id - cleanup of net.yacy.cora.document package and refactoring
2013-09-15 00:30:23 +02:00
DigestURL locationURL;
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
try {
// try to parse url
- the webgraph shall store all links which appear on a web page and not all unique links! This made it necessary, that a large portion of the parser and link processing classes must be adopted to carry a different type of link collection which carry a property attribute which are attached to web anchors. - introduction of a new URL class, AnchorURL - the other url classes, DigestURI and MultiProtocolURI had been renamed and refactored to fit into a new document package schema, document.id - cleanup of net.yacy.cora.document package and refactoring
2013-09-15 00:30:23 +02:00
locationURL = new DigestURL(location);
Added 'final' for all exception blocks as this helps the Java compiler to optimize memory usage Conflicts: source/net/yacy/search/Switchboard.java
2013-07-17 18:31:30 +02:00
} catch (final MalformedURLException e ) {
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
break;
}
PublicKey publicKey = null;
// get public key if it's in config
try {
final String publicKeyString =
getConfig("network.unit.update.location" + i + ".key", null);
if ( publicKeyString != null ) {
final byte[] publicKeyBytes =
Base64Order.standardCoder.decode(publicKeyString.trim());
publicKey = cryptoLib.getPublicKeyFromBytes(publicKeyBytes);
}
Added 'final' for all exception blocks as this helps the Java compiler to optimize memory usage Conflicts: source/net/yacy/search/Switchboard.java
2013-07-17 18:31:30 +02:00
} catch (final InvalidKeySpecException e ) {
- refactoring of log to ConcurrentLog: jdk-based logger tend to block at java.util.logging.Logger.log(Logger.java:476) in concurrent environments. This makes logging a main performance issue. To overcome this problem, this is a add-on to jdk logging to put log entries on a concurrent message queue and log the messages one by one using a separate process. - FTPClient uses the concurrent logging instead of the log4j logger
2013-07-09 14:28:25 +02:00
ConcurrentLog.logException(e);
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
}
final yacyUpdateLocation updateLocation = new yacyUpdateLocation(locationURL, publicKey);
yacyRelease.latestReleaseLocations.add(updateLocation);
i++;
}
} catch ( final NoSuchAlgorithmException e1 ) {
// TODO Auto-generated catch block
- refactoring of log to ConcurrentLog: jdk-based logger tend to block at java.util.logging.Logger.log(Logger.java:476) in concurrent environments. This makes logging a main performance issue. To overcome this problem, this is a add-on to jdk logging to put log entries on a concurrent message queue and log the messages one by one using a separate process. - FTPClient uses the concurrent logging instead of the log4j logger
2013-07-09 14:28:25 +02:00
ConcurrentLog.logException(e1);
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
}
// set white/blacklists
this.networkWhitelist = Domains.makePatterns(getConfig(SwitchboardConstants.NETWORK_WHITELIST, ""));
this.networkBlacklist = Domains.makePatterns(getConfig(SwitchboardConstants.NETWORK_BLACKLIST, ""));
/*
// in intranet and portal network set robinson mode
if (networkUnitDefinition.equals("defaults/yacy.network.webportal.unit") ||
networkUnitDefinition.equals("defaults/yacy.network.intranet.unit")) {
// switch to robinson mode
setConfig("crawlResponse", "false");
setConfig(plasmaSwitchboardConstants.INDEX_DIST_ALLOW, false);
setConfig(plasmaSwitchboardConstants.INDEX_RECEIVE_ALLOW, false);
}
// in freeworld network set full p2p mode
if (networkUnitDefinition.equals("defaults/yacy.network.freeworld.unit")) {
// switch to robinson mode
setConfig("crawlResponse", "true");
setConfig(plasmaSwitchboardConstants.INDEX_DIST_ALLOW, true);
setConfig(plasmaSwitchboardConstants.INDEX_RECEIVE_ALLOW, true);
}
*/
// write the YaCy network identification inside the yacybot client user agent to distinguish networks
Redesign of crawler identification and robots steering. A non-p2p user in intranets and the internet can now choose to appear as Googlebot. This is an essential necessity to be able to compete in the field of commercial search appliances, since most web pages are these days optimized only for Google and no other search platform any more. All commercial search engine providers have a built-in fake-Google User Agent to be able to get the same search index as Google can do. Without the resistance against obeying to robots.txt in this case, no competition is possible any more. YaCy will always obey the robots.txt when it is used for crawling the web in a peer-to-peer network, but to establish a Search Appliance (like a Google Search Appliance, GSA) it is necessary to be able to behave exactly like a Google crawler. With this change, you will be able to switch the user agent when portal or intranet mode is selected on per-crawl-start basis. Every crawl start can have a different user agent.
2013-08-22 14:23:47 +02:00
ClientIdentification.generateYaCyBot(getConfig(SwitchboardConstants.NETWORK_NAME, "")
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
+ (isRobinsonMode() ? "-" : "/")
+ getConfig(SwitchboardConstants.NETWORK_DOMAIN, "global"));
}
public void switchNetwork(final String networkDefinition) throws FileNotFoundException, IOException {
- refactoring of log to ConcurrentLog: jdk-based logger tend to block at java.util.logging.Logger.log(Logger.java:476) in concurrent environments. This makes logging a main performance issue. To overcome this problem, this is a add-on to jdk logging to put log entries on a concurrent message queue and log the messages one by one using a separate process. - FTPClient uses the concurrent logging instead of the log4j logger
2013-07-09 14:28:25 +02:00
this.log.info("SWITCH NETWORK: switching to '" + networkDefinition + "'");
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
// pause crawls
final boolean lcp = crawlJobIsPaused(SwitchboardConstants.CRAWLJOB_LOCAL_CRAWL);
if ( !lcp ) {
pauseCrawlJob(SwitchboardConstants.CRAWLJOB_LOCAL_CRAWL, "network switch to " + networkDefinition);
}
final boolean rcp = crawlJobIsPaused(SwitchboardConstants.CRAWLJOB_REMOTE_TRIGGERED_CRAWL);
if ( !rcp ) {
pauseCrawlJob(SwitchboardConstants.CRAWLJOB_REMOTE_TRIGGERED_CRAWL, "network switch to " + networkDefinition);
}
// trigger online caution
this.proxyLastAccess = System.currentTimeMillis() + 3000; // at least 3 seconds online caution to prevent unnecessary action on database meanwhile
- refactoring of log to ConcurrentLog: jdk-based logger tend to block at java.util.logging.Logger.log(Logger.java:476) in concurrent environments. This makes logging a main performance issue. To overcome this problem, this is a add-on to jdk logging to put log entries on a concurrent message queue and log the messages one by one using a separate process. - FTPClient uses the concurrent logging instead of the log4j logger
2013-07-09 14:28:25 +02:00
this.log.info("SWITCH NETWORK: SHUT DOWN OF OLD INDEX DATABASE...");
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
// clean search events which have cached relations to the old index
SearchEventCache.cleanupEvents(true);
// switch the networks
synchronized ( this ) {
// remember the solr scheme
introduced a second core named 'webgraph'. This core will hold the link structure, but is not filled yet. To have the opportunity of a second core, multi-core functionality had to be implemented to the deep-embedded solr: - migrated the solr_40 directory content to a subdirectory 'collection1'; the previously used default core is now called collection1 - added solr_40/webgraph subdirectory as second core - added a servlet configuration for the second core 'webgraph' in /IndexSchema_p.html - added instance handling as addition to solr connections: all solr connectors are now instances of an solr 'instance' object; this required a complete re-design of the solr embedding - migrated also caching and sharding ontop of new instance handling - migrated the search apis to handle now the access to a specific core, the default core named 'collection1' - migrated the remote solr search interface to access shards of cores; for the yacy remote search the default core is now called 'solr'; using the peer address as solr address - migrated the solr backup and restore process: old backups cannot be used after this migration! - redesign of solr instance handling in all methods which access the instances: they cannot hold copies of these instances any more; the must retrieve the actuall connection object every time they want to write to it (this solves also some bugs when switching the index/network) - added another schema 'solr.webgraph.schema', the old solr.keys.list is replaced by solr.collection.schema
2013-02-21 13:23:55 +01:00
CollectionConfiguration collectionConfiguration = this.index.fulltext().getDefaultConfiguration();
WebgraphConfiguration webgraphConfiguration = this.index.fulltext().getWebgraphConfiguration();
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
// shut down
this.crawler.close();
if ( this.dhtDispatcher != null ) {
this.dhtDispatcher.close();
}
synchronized ( this.index ) {
this.index.close();
}
this.crawlStacker.announceClose();
this.crawlStacker.close();
this.webStructure.close();
- refactoring of log to ConcurrentLog: jdk-based logger tend to block at java.util.logging.Logger.log(Logger.java:476) in concurrent environments. This makes logging a main performance issue. To overcome this problem, this is a add-on to jdk logging to put log entries on a concurrent message queue and log the messages one by one using a separate process. - FTPClient uses the concurrent logging instead of the log4j logger
2013-07-09 14:28:25 +02:00
this.log.info("SWITCH NETWORK: START UP OF NEW INDEX DATABASE...");
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
// new properties
setConfig("network.unit.definition", networkDefinition);
overwriteNetworkDefinition();
final File indexPrimaryPath =
getDataPath(SwitchboardConstants.INDEX_PRIMARY_PATH, SwitchboardConstants.INDEX_PATH_DEFAULT);
final int wordCacheMaxCount =
(int) getConfigLong(SwitchboardConstants.WORDCACHE_MAX_COUNT, 20000);
final long fileSizeMax =
(OS.isWindows) ? this.getConfigLong("filesize.max.win", Integer.MAX_VALUE) : this.getConfigLong("filesize.max.other", Integer.MAX_VALUE);
final int redundancy = (int) this.getConfigLong("network.unit.dhtredundancy.senior", 1);
final int partitionExponent = (int) this.getConfigLong("network.unit.dht.partitionExponent", 0);
final String networkName = getConfig(SwitchboardConstants.NETWORK_NAME, "");
this.networkRoot = new File(new File(indexPrimaryPath, networkName), "NETWORK");
this.queuesRoot = new File(new File(indexPrimaryPath, networkName), "QUEUES");
this.networkRoot.mkdirs();
this.queuesRoot.mkdirs();
// clear statistic data
ResultURLs.clearStacks();
// remove heuristics
added a 'greedy learning' mechanismn which will cause that a 'fresh' yacy will load linked web pages from search results until the total number of web pages reaches 15000. This shall give fresh peers a 'boost' to get faster a personalized search index.
2013-06-11 14:42:30 +02:00
setConfig(SwitchboardConstants.HEURISTIC_SITE, false);
setConfig(SwitchboardConstants.HEURISTIC_BLEKKO, false);
setConfig(SwitchboardConstants.HEURISTIC_TWITTER, false);
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
// relocate
this.peers.relocate(
this.networkRoot,
redundancy,
partitionExponent,
this.useTailCache,
this.exceed134217727);
introduced a second core named 'webgraph'. This core will hold the link structure, but is not filled yet. To have the opportunity of a second core, multi-core functionality had to be implemented to the deep-embedded solr: - migrated the solr_40 directory content to a subdirectory 'collection1'; the previously used default core is now called collection1 - added solr_40/webgraph subdirectory as second core - added a servlet configuration for the second core 'webgraph' in /IndexSchema_p.html - added instance handling as addition to solr connections: all solr connectors are now instances of an solr 'instance' object; this required a complete re-design of the solr embedding - migrated also caching and sharding ontop of new instance handling - migrated the search apis to handle now the access to a specific core, the default core named 'collection1' - migrated the remote solr search interface to access shards of cores; for the yacy remote search the default core is now called 'solr'; using the peer address as solr address - migrated the solr backup and restore process: old backups cannot be used after this migration! - redesign of solr instance handling in all methods which access the instances: they cannot hold copies of these instances any more; the must retrieve the actuall connection object every time they want to write to it (this solves also some bugs when switching the index/network) - added another schema 'solr.webgraph.schema', the old solr.keys.list is replaced by solr.collection.schema
2013-02-21 13:23:55 +01:00
this.index = new Segment(this.log, new File(new File(indexPrimaryPath, networkName), "SEGMENTS"), collectionConfiguration, webgraphConfiguration);
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
if (this.getConfigBool(SwitchboardConstants.CORE_SERVICE_RWI, true)) this.index.connectRWI(wordCacheMaxCount, fileSizeMax);
if (this.getConfigBool(SwitchboardConstants.CORE_SERVICE_CITATION, true)) this.index.connectCitation(wordCacheMaxCount, fileSizeMax);
if (this.getConfigBool(SwitchboardConstants.CORE_SERVICE_FULLTEXT, true)) {
removed the commitWithin attribute because that is not the way how the index is updated the right way for us. May also be be superfluous with the solr 4.0 softcommit.
2013-02-13 02:29:47 +01:00
this.index.fulltext().connectLocalSolr();
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
this.index.connectUrlDb(this.useTailCache, this.exceed134217727);
}
enhanced webgraph processing
2013-12-04 01:54:45 +01:00
this.index.fulltext().setUseWebgraph(this.getConfigBool(SwitchboardConstants.CORE_SERVICE_WEBGRAPH, false));
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
// set up the solr interface
final String solrurls = getConfig(SwitchboardConstants.FEDERATED_SERVICE_SOLR_INDEXING_URL, "http://127.0.0.1:8983/solr");
final boolean usesolr = getConfigBool(SwitchboardConstants.FEDERATED_SERVICE_SOLR_INDEXING_ENABLED, false) & solrurls.length() > 0;
tried to fix timeout and connection-lost problems when using an outside solr.
2013-11-28 01:31:53 +01:00
final int solrtimeout = getConfigInt(SwitchboardConstants.FEDERATED_SERVICE_SOLR_INDEXING_TIMEOUT, 60000);
added option to attach remote solr servers in read-only mode
2013-12-27 02:55:21 +01:00
final boolean writeEnabled = getConfigBool(SwitchboardConstants.FEDERATED_SERVICE_SOLR_INDEXING_WRITEENABLED, true);
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
if (usesolr && solrurls != null && solrurls.length() > 0) {
try {
added the property federated.service.solr.indexing.timeout to yacy.init to provide a configurable time-out for solr; see also: http://bugs.yacy.net/view.php?id=254
2013-07-22 17:45:12 +02:00
ArrayList<RemoteInstance> instances = RemoteInstance.getShardInstances(solrurls, null, null, solrtimeout);
added option to attach remote solr servers in read-only mode
2013-12-27 02:55:21 +01:00
this.index.fulltext().connectRemoteSolr(instances, writeEnabled);
Added 'final' for all exception blocks as this helps the Java compiler to optimize memory usage Conflicts: source/net/yacy/search/Switchboard.java
2013-07-17 18:31:30 +02:00
} catch (final IOException e ) {
- refactoring of log to ConcurrentLog: jdk-based logger tend to block at java.util.logging.Logger.log(Logger.java:476) in concurrent environments. This makes logging a main performance issue. To overcome this problem, this is a add-on to jdk logging to put log entries on a concurrent message queue and log the messages one by one using a separate process. - FTPClient uses the concurrent logging instead of the log4j logger
2013-07-09 14:28:25 +02:00
ConcurrentLog.logException(e);
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
}
}
// create a crawler
this.crawlQueues.relocate(this.queuesRoot); // cannot be closed because the busy threads are working with that object
- crawl profiles shall be deleted from active and passive stacks if they are deleted to terminate the crawl because otherwise the crawl will go on after the load-from-passive stack policy. - better check if a crawl is terminated using the loader queue.
2013-09-26 10:22:31 +02:00
this.crawler = new CrawlSwitchboard(networkName, this);
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
// init a DHT transmission dispatcher
this.dhtDispatcher =
(this.peers.sizeConnected() == 0) ? null : new Dispatcher(
this.index,
this.peers,
true,
10000);
// create new web structure
this.webStructure = new WebStructureGraph(new File(this.queuesRoot, "webStructure.map"));
// load domainList
try {
this.domainList = null;
if ( !getConfig("network.unit.domainlist", "").equals("") ) {
final Reader r =
getConfigFileFromWebOrLocally(getConfig("network.unit.domainlist", ""), getAppPath()
.getAbsolutePath(), new File(this.networkRoot, "domainlist.txt"));
this.domainList = new FilterEngine();
this.domainList.loadList(new BufferedReader(r), null);
}
Added 'final' for all exception blocks as this helps the Java compiler to optimize memory usage Conflicts: source/net/yacy/search/Switchboard.java
2013-07-17 18:31:30 +02:00
} catch (final FileNotFoundException e ) {
- refactoring of log to ConcurrentLog: jdk-based logger tend to block at java.util.logging.Logger.log(Logger.java:476) in concurrent environments. This makes logging a main performance issue. To overcome this problem, this is a add-on to jdk logging to put log entries on a concurrent message queue and log the messages one by one using a separate process. - FTPClient uses the concurrent logging instead of the log4j logger
2013-07-09 14:28:25 +02:00
this.log.severe("CONFIG: domainlist not found: " + e.getMessage());
Added 'final' for all exception blocks as this helps the Java compiler to optimize memory usage Conflicts: source/net/yacy/search/Switchboard.java
2013-07-17 18:31:30 +02:00
} catch (final IOException e ) {
- refactoring of log to ConcurrentLog: jdk-based logger tend to block at java.util.logging.Logger.log(Logger.java:476) in concurrent environments. This makes logging a main performance issue. To overcome this problem, this is a add-on to jdk logging to put log entries on a concurrent message queue and log the messages one by one using a separate process. - FTPClient uses the concurrent logging instead of the log4j logger
2013-07-09 14:28:25 +02:00
this.log.severe("CONFIG: error while retrieving domainlist: " + e.getMessage());
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
}
this.crawlStacker =
new CrawlStacker(
this.robots,
this.crawlQueues,
this.crawler,
this.index,
this.peers,
"local.any".indexOf(getConfig(SwitchboardConstants.NETWORK_DOMAIN, "global")) >= 0,
"global.any".indexOf(getConfig(SwitchboardConstants.NETWORK_DOMAIN, "global")) >= 0,
this.domainList);
}
Domains.setNoLocalCheck(isAllIPMode()); // possibly switch off localIP check
// start up crawl jobs
continueCrawlJob(SwitchboardConstants.CRAWLJOB_LOCAL_CRAWL);
continueCrawlJob(SwitchboardConstants.CRAWLJOB_REMOTE_TRIGGERED_CRAWL);
this.log
- refactoring of log to ConcurrentLog: jdk-based logger tend to block at java.util.logging.Logger.log(Logger.java:476) in concurrent environments. This makes logging a main performance issue. To overcome this problem, this is a add-on to jdk logging to put log entries on a concurrent message queue and log the messages one by one using a separate process. - FTPClient uses the concurrent logging instead of the log4j logger
2013-07-09 14:28:25 +02:00
.info("SWITCH NETWORK: FINISHED START UP, new network is now '" + networkDefinition + "'.");
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
// set the network-specific remote crawl ppm
setRemotecrawlPPM(Math.max(1, (int) getConfigLong("network.unit.remotecrawl.speed", 60)));
}
public void setRemotecrawlPPM(final int ppm) {
final long newBusySleep = Math.max(100, 60000 / ppm);
// propagate to crawler
final BusyThread rct = getThread(SwitchboardConstants.CRAWLJOB_REMOTE_TRIGGERED_CRAWL);
setConfig(SwitchboardConstants.CRAWLJOB_REMOTE_TRIGGERED_CRAWL_BUSYSLEEP, newBusySleep);
setConfig(
SwitchboardConstants.CRAWLJOB_REMOTE_TRIGGERED_CRAWL_IDLESLEEP,
Math.min(10000, newBusySleep * 10));
rct.setBusySleep(getConfigLong(SwitchboardConstants.CRAWLJOB_REMOTE_TRIGGERED_CRAWL_BUSYSLEEP, 1000));
rct
.setIdleSleep(getConfigLong(SwitchboardConstants.CRAWLJOB_REMOTE_TRIGGERED_CRAWL_IDLESLEEP, 10000));
// propagate to loader
final BusyThread rcl = getThread(SwitchboardConstants.CRAWLJOB_REMOTE_CRAWL_LOADER);
setConfig(SwitchboardConstants.CRAWLJOB_REMOTE_CRAWL_LOADER_BUSYSLEEP, newBusySleep * 4);
setConfig(
SwitchboardConstants.CRAWLJOB_REMOTE_CRAWL_LOADER_IDLESLEEP,
Math.min(10000, newBusySleep * 20));
rcl.setBusySleep(getConfigLong(SwitchboardConstants.CRAWLJOB_REMOTE_CRAWL_LOADER_BUSYSLEEP, 1000));
rcl.setIdleSleep(getConfigLong(SwitchboardConstants.CRAWLJOB_REMOTE_CRAWL_LOADER_IDLESLEEP, 10000));
}
public void initMessages() throws IOException {
- refactoring of log to ConcurrentLog: jdk-based logger tend to block at java.util.logging.Logger.log(Logger.java:476) in concurrent environments. This makes logging a main performance issue. To overcome this problem, this is a add-on to jdk logging to put log entries on a concurrent message queue and log the messages one by one using a separate process. - FTPClient uses the concurrent logging instead of the log4j logger
2013-07-09 14:28:25 +02:00
this.log.config("Starting Message Board");
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
final File messageDbFile = new File(this.workPath, "message.heap");
this.messageDB = new MessageBoard(messageDbFile);
- refactoring of log to ConcurrentLog: jdk-based logger tend to block at java.util.logging.Logger.log(Logger.java:476) in concurrent environments. This makes logging a main performance issue. To overcome this problem, this is a add-on to jdk logging to put log entries on a concurrent message queue and log the messages one by one using a separate process. - FTPClient uses the concurrent logging instead of the log4j logger
2013-07-09 14:28:25 +02:00
this.log.config("Loaded Message Board DB from file "
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
+ messageDbFile.getName()
+ ", "
+ this.messageDB.size()
+ " entries"
+ ", "
+ ppRamString(messageDbFile.length() / 1024));
}
public void initWiki() throws IOException {
- refactoring of log to ConcurrentLog: jdk-based logger tend to block at java.util.logging.Logger.log(Logger.java:476) in concurrent environments. This makes logging a main performance issue. To overcome this problem, this is a add-on to jdk logging to put log entries on a concurrent message queue and log the messages one by one using a separate process. - FTPClient uses the concurrent logging instead of the log4j logger
2013-07-09 14:28:25 +02:00
this.log.config("Starting Wiki Board");
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
final File wikiDbFile = new File(this.workPath, "wiki.heap");
this.wikiDB = new WikiBoard(wikiDbFile, new File(this.workPath, "wiki-bkp.heap"));
- refactoring of log to ConcurrentLog: jdk-based logger tend to block at java.util.logging.Logger.log(Logger.java:476) in concurrent environments. This makes logging a main performance issue. To overcome this problem, this is a add-on to jdk logging to put log entries on a concurrent message queue and log the messages one by one using a separate process. - FTPClient uses the concurrent logging instead of the log4j logger
2013-07-09 14:28:25 +02:00
this.log.config("Loaded Wiki Board DB from file "
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
+ wikiDbFile.getName()
+ ", "
+ this.wikiDB.size()
+ " entries"
+ ", "
+ ppRamString(wikiDbFile.length() / 1024));
}
public void initBlog() throws IOException {
- refactoring of log to ConcurrentLog: jdk-based logger tend to block at java.util.logging.Logger.log(Logger.java:476) in concurrent environments. This makes logging a main performance issue. To overcome this problem, this is a add-on to jdk logging to put log entries on a concurrent message queue and log the messages one by one using a separate process. - FTPClient uses the concurrent logging instead of the log4j logger
2013-07-09 14:28:25 +02:00
this.log.config("Starting Blog");
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
final File blogDbFile = new File(this.workPath, "blog.heap");
this.blogDB = new BlogBoard(blogDbFile);
- refactoring of log to ConcurrentLog: jdk-based logger tend to block at java.util.logging.Logger.log(Logger.java:476) in concurrent environments. This makes logging a main performance issue. To overcome this problem, this is a add-on to jdk logging to put log entries on a concurrent message queue and log the messages one by one using a separate process. - FTPClient uses the concurrent logging instead of the log4j logger
2013-07-09 14:28:25 +02:00
this.log.config("Loaded Blog DB from file "
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
+ blogDbFile.getName()
+ ", "
+ this.blogDB.size()
+ " entries"
+ ", "
+ ppRamString(blogDbFile.length() / 1024));
final File blogCommentDbFile = new File(this.workPath, "blogComment.heap");
this.blogCommentDB = new BlogBoardComments(blogCommentDbFile);
- refactoring of log to ConcurrentLog: jdk-based logger tend to block at java.util.logging.Logger.log(Logger.java:476) in concurrent environments. This makes logging a main performance issue. To overcome this problem, this is a add-on to jdk logging to put log entries on a concurrent message queue and log the messages one by one using a separate process. - FTPClient uses the concurrent logging instead of the log4j logger
2013-07-09 14:28:25 +02:00
this.log.config("Loaded Blog-Comment DB from file "
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
+ blogCommentDbFile.getName()
+ ", "
+ this.blogCommentDB.size()
+ " entries"
+ ", "
+ ppRamString(blogCommentDbFile.length() / 1024));
}
public void initBookmarks() throws IOException {
- refactoring of log to ConcurrentLog: jdk-based logger tend to block at java.util.logging.Logger.log(Logger.java:476) in concurrent environments. This makes logging a main performance issue. To overcome this problem, this is a add-on to jdk logging to put log entries on a concurrent message queue and log the messages one by one using a separate process. - FTPClient uses the concurrent logging instead of the log4j logger
2013-07-09 14:28:25 +02:00
this.log.config("Loading Bookmarks DB");
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
final File bookmarksFile = new File(this.workPath, "bookmarks.heap");
final File tagsFile = new File(this.workPath, "bookmarkTags.heap");
final File datesFile = new File(this.workPath, "bookmarkDates.heap");
tagsFile.delete();
this.bookmarksDB = new BookmarksDB(bookmarksFile, datesFile);
- refactoring of log to ConcurrentLog: jdk-based logger tend to block at java.util.logging.Logger.log(Logger.java:476) in concurrent environments. This makes logging a main performance issue. To overcome this problem, this is a add-on to jdk logging to put log entries on a concurrent message queue and log the messages one by one using a separate process. - FTPClient uses the concurrent logging instead of the log4j logger
2013-07-09 14:28:25 +02:00
this.log.config("Loaded Bookmarks DB from files "
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
+ bookmarksFile.getName()
+ ", "
+ tagsFile.getName());
- refactoring of log to ConcurrentLog: jdk-based logger tend to block at java.util.logging.Logger.log(Logger.java:476) in concurrent environments. This makes logging a main performance issue. To overcome this problem, this is a add-on to jdk logging to put log entries on a concurrent message queue and log the messages one by one using a separate process. - FTPClient uses the concurrent logging instead of the log4j logger
2013-07-09 14:28:25 +02:00
this.log.config(this.bookmarksDB.tagsSize()
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
+ " Tag, "
+ this.bookmarksDB.bookmarksSize()
+ " Bookmarks");
}
public static Switchboard getSwitchboard() {
return sb;
}
Redesign of crawler identification and robots steering. A non-p2p user in intranets and the internet can now choose to appear as Googlebot. This is an essential necessity to be able to compete in the field of commercial search appliances, since most web pages are these days optimized only for Google and no other search platform any more. All commercial search engine providers have a built-in fake-Google User Agent to be able to get the same search index as Google can do. Without the resistance against obeying to robots.txt in this case, no competition is possible any more. YaCy will always obey the robots.txt when it is used for crawling the web in a peer-to-peer network, but to establish a Search Appliance (like a Google Search Appliance, GSA) it is necessary to be able to behave exactly like a Google crawler. With this change, you will be able to switch the user agent when portal or intranet mode is selected on per-crawl-start basis. Every crawl start can have a different user agent.
2013-08-22 14:23:47 +02:00
public boolean isP2PMode() {
return getConfig(SwitchboardConstants.NETWORK_BOOTSTRAP_SEEDLIST_STUB + "0", null) != null;
}
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
public boolean isIntranetMode() {
return "local.any".indexOf(getConfig(SwitchboardConstants.NETWORK_DOMAIN, "global")) >= 0;
}
public boolean isGlobalMode() {
return "global.any".indexOf(getConfig(SwitchboardConstants.NETWORK_DOMAIN, "global")) >= 0;
}
public boolean isAllIPMode() {
return "any".indexOf(getConfig(SwitchboardConstants.NETWORK_DOMAIN, "global")) >= 0;
}
/**
* in nocheck mode the isLocal property is not checked to omit DNS lookup. Can only be done in allip mode
*
* @return
*/
public boolean isIPNoCheckMode() {
return isAllIPMode() && getConfigBool(SwitchboardConstants.NETWORK_DOMAIN_NOCHECK, false);
}
public boolean isRobinsonMode() {
// we are in robinson mode, if we do not exchange index by dht distribution
// we need to take care that search requests and remote indexing requests go only
// to the peers in the same cluster, if we run a robinson cluster.
return (this.peers != null && this.peers.sizeConnected() == 0)
|| (!getConfigBool(SwitchboardConstants.INDEX_DIST_ALLOW, false) && !getConfigBool(
SwitchboardConstants.INDEX_RECEIVE_ALLOW,
false));
}
public boolean isPublicRobinson() {
// robinson peers may be member of robinson clusters, which can be public or private
// this does not check the robinson attribute, only the specific subtype of the cluster
final String clustermode =
getConfig(SwitchboardConstants.CLUSTER_MODE, SwitchboardConstants.CLUSTER_MODE_PUBLIC_PEER);
return (clustermode.equals(SwitchboardConstants.CLUSTER_MODE_PUBLIC_CLUSTER))
|| (clustermode.equals(SwitchboardConstants.CLUSTER_MODE_PUBLIC_PEER));
}
public boolean isInMyCluster(final String peer) {
// check if the given peer is in the own network, if this is a robinson cluster
// depending on the robinson cluster type, the peer String may be a peerhash (b64-hash)
// or a ip:port String or simply a ip String
// if this robinson mode does not define a cluster membership, false is returned
if ( peer == null ) {
return false;
}
if ( !isRobinsonMode() ) {
return false;
}
final String clustermode =
getConfig(SwitchboardConstants.CLUSTER_MODE, SwitchboardConstants.CLUSTER_MODE_PUBLIC_PEER);
if ( clustermode.equals(SwitchboardConstants.CLUSTER_MODE_PUBLIC_CLUSTER) ) {
// check if we got the request from a peer in the public cluster
return this.clusterhashes.containsKey(ASCII.getBytes(peer));
}
return false;
}
public boolean isInMyCluster(final Seed seed) {
// check if the given peer is in the own network, if this is a robinson cluster
// if this robinson mode does not define a cluster membership, false is returned
if ( seed == null ) {
return false;
}
if ( !isRobinsonMode() ) {
return false;
}
final String clustermode =
getConfig(SwitchboardConstants.CLUSTER_MODE, SwitchboardConstants.CLUSTER_MODE_PUBLIC_PEER);
if ( clustermode.equals(SwitchboardConstants.CLUSTER_MODE_PUBLIC_CLUSTER) ) {
// check if we got the request from a peer in the public cluster
return this.clusterhashes.containsKey(ASCII.getBytes(seed.hash));
}
return false;
}
redesign of index.exist-test: this shall now not be done using a single id to be tested, but with a collection of ids. This will cause only a single call to solr instead of many. The result is a much better performace when testing the existence of many urls. The effect should cause very much less IO during index transmission, both on sender and receiver side.
2013-05-17 13:59:37 +02:00
/**
* tests if hash occurs in any database.
* @param hash
* @return if it exists, the name of the database is returned, if it not exists, null is returned
*/
@Deprecated
added the generation of 50 (!!) new solr field in the core 'webgraph'. The default schema uses only some of them and the resting search index has now the following properties: - webgraph size will have about 40 times as much entries as default index - the complete index size will increase and may be about the double size of current amount As testing showed, not much indexing performance is lost. The default index will be smaller (moved fields out of it); thus searching can be faster. The new index will cause that some old parts in YaCy can be removed, i.e. specialized webgraph data and the noload crawler. The new index will make it possible to: - search within link texts of linked but not indexed documents (about 20 times of document index in size!!) - get a very detailed link graph - enhance ranking using a complete link graph To get the full access to the new index, the API to solr has now two access points: one with attribute core=collection1 for the default search index and core=webgraph to the new webgraph search index. This is also avaiable for p2p operation but client access is not yet implemented.
2013-02-22 15:45:15 +01:00
public HarvestProcess urlExists(final String hash) {
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
if (this.index.exists(hash)) return HarvestProcess.LOADED;
refactoring (im preparation for new removeHost method)
2013-09-05 09:59:41 +02:00
return this.crawlQueues.exists(ASCII.getBytes(hash));
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
}
redesign of index.exist-test: this shall now not be done using a single id to be tested, but with a collection of ids. This will cause only a single call to solr instead of many. The result is a much better performace when testing the existence of many urls. The effect should cause very much less IO during index transmission, both on sender and receiver side.
2013-05-17 13:59:37 +02:00
/**
* tests if hashes occur in any database.
* @param ids a collection of url hashes
* @return a map from the hash id to: if it exists, the name of the database, otherwise null
*/
less overhead calling exist() with only one hash
2013-11-04 09:37:31 +01:00
public Map<String, HarvestProcess> urlExists(final Set<String> ids) {
redesign of index.exist-test: this shall now not be done using a single id to be tested, but with a collection of ids. This will cause only a single call to solr instead of many. The result is a much better performace when testing the existence of many urls. The effect should cause very much less IO during index transmission, both on sender and receiver side.
2013-05-17 13:59:37 +02:00
Set<String> e = this.index.exists(ids);
Map<String, HarvestProcess> m = new HashMap<String, HarvestProcess>();
for (String id: ids) {
refactoring (im preparation for new removeHost method)
2013-09-05 09:59:41 +02:00
m.put(id, e.contains(id) ? HarvestProcess.LOADED : this.crawlQueues.exists(ASCII.getBytes(id)));
redesign of index.exist-test: this shall now not be done using a single id to be tested, but with a collection of ids. This will cause only a single call to solr instead of many. The result is a much better performace when testing the existence of many urls. The effect should cause very much less IO during index transmission, both on sender and receiver side.
2013-05-17 13:59:37 +02:00
}
return m;
}
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
public void urlRemove(final Segment segment, final byte[] hash) {
segment.fulltext().remove(hash);
ResultURLs.remove(ASCII.String(hash));
refactoring (im preparation for new removeHost method)
2013-09-05 09:59:41 +02:00
this.crawlQueues.removeURL(hash);
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
}
- the webgraph shall store all links which appear on a web page and not all unique links! This made it necessary, that a large portion of the parser and link processing classes must be adopted to carry a different type of link collection which carry a property attribute which are attached to web anchors. - introduction of a new URL class, AnchorURL - the other url classes, DigestURI and MultiProtocolURI had been renamed and refactored to fit into a new document package schema, document.id - cleanup of net.yacy.cora.document package and refactoring
2013-09-15 00:30:23 +02:00
public DigestURL getURL(final byte[] urlhash) {
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
if (urlhash == null) return null;
if (urlhash.length == 0) return null;
- the webgraph shall store all links which appear on a web page and not all unique links! This made it necessary, that a large portion of the parser and link processing classes must be adopted to carry a different type of link collection which carry a property attribute which are attached to web anchors. - introduction of a new URL class, AnchorURL - the other url classes, DigestURI and MultiProtocolURI had been renamed and refactored to fit into a new document package schema, document.id - cleanup of net.yacy.cora.document package and refactoring
2013-09-15 00:30:23 +02:00
final DigestURL url = this.index.fulltext().getURL(urlhash);
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
if (url != null) return url;
return this.crawlQueues.getURL(urlhash);
}
public RankingProfile getRanking() {
return (getConfig(SwitchboardConstants.SEARCH_RANKING_RWI_PROFILE, "").isEmpty())
? new RankingProfile(Classification.ContentDomain.TEXT)
: new RankingProfile("", crypt.simpleDecode(this.getConfig(SwitchboardConstants.SEARCH_RANKING_RWI_PROFILE, "")));
}
/**
* checks if the proxy, the local search or remote search was accessed some time before If no limit is
* exceeded, null is returned. If a limit is exceeded, then the name of the service that caused the
* caution is returned
*
* @return
*/
public String onlineCaution() {
if ( System.currentTimeMillis() - this.proxyLastAccess < Integer.parseInt(getConfig(
SwitchboardConstants.PROXY_ONLINE_CAUTION_DELAY,
"100")) ) {
return "proxy";
}
if ( System.currentTimeMillis() - this.localSearchLastAccess < Integer.parseInt(getConfig(
SwitchboardConstants.LOCALSEACH_ONLINE_CAUTION_DELAY,
"1000")) ) {
return "localsearch";
}
if ( System.currentTimeMillis() - this.remoteSearchLastAccess < Integer.parseInt(getConfig(
SwitchboardConstants.REMOTESEARCH_ONLINE_CAUTION_DELAY,
"500")) ) {
return "remotesearch";
}
return null;
}
/**
* Creates a human readable string from a number which represents the size of a file. The method has a
* side effect: It changes the input. Since the method is private and only used in this class for values
* which are not used later, this should be OK in this case, but one should never use this method without
* thinking about the side effect. [MN]
*
* @param bytes the length of a file
* @return the length of a file in human readable form
*/
private static String ppRamString(long bytes) {
if ( bytes < 1024 ) {
return bytes + " KByte";
}
bytes = bytes / 1024;
if ( bytes < 1024 ) {
return bytes + " MByte";
}
bytes = bytes / 1024;
if ( bytes < 1024 ) {
return bytes + " GByte";
}
return (bytes / 1024) + "TByte";
}
/**
* {@link CrawlProfiles Crawl Profiles} are saved independently from the queues themselves and therefore
* have to be cleaned up from time to time. This method only performs the clean-up if - and only if - the
* {@link IndexingStack switchboard}, {@link LoaderDispatcher loader} and {@link plasmaCrawlNURL local
* crawl} queues are all empty.
* <p>
* Then it iterates through all existing {@link CrawlProfiles crawl profiles} and removes all profiles
* which are not hard-coded.
* </p>
* <p>
* <i>If this method encounters DB-failures, the profile DB will be reseted and</i> <code>true</code><i>
* will be returned</i>
* </p>
*
* @see #CRAWL_PROFILE_PROXY hardcoded
* @see #CRAWL_PROFILE_REMOTE hardcoded
* @see #CRAWL_PROFILE_SNIPPET_TEXT hardcoded
* @see #CRAWL_PROFILE_SNIPPET_MEDIA hardcoded
* @return whether this method has done something or not (i.e. because the queues have been filled or
* there are no profiles left to clean up)
* @throws <b>InterruptedException</b> if the current thread has been interrupted, i.e. by the shutdown
* procedure
*/
public boolean cleanProfiles() throws InterruptedException {
if (getIndexingProcessorsQueueSize() > 0 ||
this.crawlQueues.workerSize() > 0 ||
this.crawlQueues.coreCrawlJobSize() > 0 ||
this.crawlQueues.limitCrawlJobSize() > 0 ||
this.crawlQueues.remoteTriggeredCrawlJobSize() > 0 ||
this.crawlQueues.noloadCrawlJobSize() > 0 ||
(this.crawlStacker != null && !this.crawlStacker.isEmpty()) ||
this.crawlQueues.noticeURL.notEmpty()) {
return false;
}
return this.crawler.clear();
}
public synchronized void close() {
- refactoring of log to ConcurrentLog: jdk-based logger tend to block at java.util.logging.Logger.log(Logger.java:476) in concurrent environments. This makes logging a main performance issue. To overcome this problem, this is a add-on to jdk logging to put log entries on a concurrent message queue and log the messages one by one using a separate process. - FTPClient uses the concurrent logging instead of the log4j logger
2013-07-09 14:28:25 +02:00
this.log.config("SWITCHBOARD SHUTDOWN STEP 1: sending termination signal to managed threads:");
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
MemoryTracker.stopSystemProfiling();
terminateAllThreads(true);
net.yacy.gui.framework.Switchboard.shutdown();
- refactoring of log to ConcurrentLog: jdk-based logger tend to block at java.util.logging.Logger.log(Logger.java:476) in concurrent environments. This makes logging a main performance issue. To overcome this problem, this is a add-on to jdk logging to put log entries on a concurrent message queue and log the messages one by one using a separate process. - FTPClient uses the concurrent logging instead of the log4j logger
2013-07-09 14:28:25 +02:00
this.log.config("SWITCHBOARD SHUTDOWN STEP 2: sending termination signal to threaded indexing");
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
// closing all still running db importer jobs
this.crawlStacker.announceClose();
this.crawlStacker.close();
this.indexingDocumentProcessor.shutdown();
this.indexingCondensementProcessor.shutdown();
this.indexingAnalysisProcessor.shutdown();
this.indexingStorageProcessor.shutdown();
if ( this.dhtDispatcher != null ) {
this.dhtDispatcher.close();
}
// de.anomic.http.client.Client.closeAllConnections();
this.wikiDB.close();
this.blogDB.close();
this.blogCommentDB.close();
this.userDB.close();
if ( this.bookmarksDB != null ) {
this.bookmarksDB.close(); // may null if concurrent initialization was not finished
}
this.messageDB.close();
this.webStructure.close();
this.crawlQueues.close();
this.crawler.close();
- refactoring of log to ConcurrentLog: jdk-based logger tend to block at java.util.logging.Logger.log(Logger.java:476) in concurrent environments. This makes logging a main performance issue. To overcome this problem, this is a add-on to jdk logging to put log entries on a concurrent message queue and log the messages one by one using a separate process. - FTPClient uses the concurrent logging instead of the log4j logger
2013-07-09 14:28:25 +02:00
this.log.config("SWITCHBOARD SHUTDOWN STEP 3: sending termination signal to database manager (stand by...)");
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
this.index.close();
this.peers.close();
Cache.close();
this.tables.close();
Domains.close();
AccessTracker.dumpLog();
Switchboard.urlBlacklist.close();
UPnP.deletePortMapping();
this.tray.remove();
try {
HTTPClient.closeConnectionManager();
Added 'final' for all exception blocks as this helps the Java compiler to optimize memory usage Conflicts: source/net/yacy/search/Switchboard.java
2013-07-17 18:31:30 +02:00
} catch (final InterruptedException e ) {
- refactoring of log to ConcurrentLog: jdk-based logger tend to block at java.util.logging.Logger.log(Logger.java:476) in concurrent environments. This makes logging a main performance issue. To overcome this problem, this is a add-on to jdk logging to put log entries on a concurrent message queue and log the messages one by one using a separate process. - FTPClient uses the concurrent logging instead of the log4j logger
2013-07-09 14:28:25 +02:00
ConcurrentLog.logException(e);
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
}
- refactoring of log to ConcurrentLog: jdk-based logger tend to block at java.util.logging.Logger.log(Logger.java:476) in concurrent environments. This makes logging a main performance issue. To overcome this problem, this is a add-on to jdk logging to put log entries on a concurrent message queue and log the messages one by one using a separate process. - FTPClient uses the concurrent logging instead of the log4j logger
2013-07-09 14:28:25 +02:00
this.log.config("SWITCHBOARD SHUTDOWN TERMINATED");
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
}
/**
* pass a response to the indexer
*
* @param response
* @return null if successful, an error message otherwise
*/
public String toIndexer(final Response response) {
assert response != null;
// get next queue entry and start a queue processing
if ( response == null ) {
if ( this.log.isFine() ) {
- refactoring of log to ConcurrentLog: jdk-based logger tend to block at java.util.logging.Logger.log(Logger.java:476) in concurrent environments. This makes logging a main performance issue. To overcome this problem, this is a add-on to jdk logging to put log entries on a concurrent message queue and log the messages one by one using a separate process. - FTPClient uses the concurrent logging instead of the log4j logger
2013-07-09 14:28:25 +02:00
this.log.fine("deQueue: queue entry is null");
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
}
return "queue entry is null";
}
if ( response.profile() == null ) {
if ( this.log.isFine() ) {
- refactoring of log to ConcurrentLog: jdk-based logger tend to block at java.util.logging.Logger.log(Logger.java:476) in concurrent environments. This makes logging a main performance issue. To overcome this problem, this is a add-on to jdk logging to put log entries on a concurrent message queue and log the messages one by one using a separate process. - FTPClient uses the concurrent logging instead of the log4j logger
2013-07-09 14:28:25 +02:00
this.log.fine("deQueue: profile is null");
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
}
return "profile is null";
}
// check if the document should be indexed based on proxy/crawler rules
String noIndexReason = "unspecified indexing error";
if ( response.processCase(this.peers.mySeed().hash) == EventOrigin.PROXY_LOAD ) {
// proxy-load
noIndexReason = response.shallIndexCacheForProxy();
} else {
// normal crawling
noIndexReason = response.shallIndexCacheForCrawler();
}
// check if the parser supports the mime type
if ( noIndexReason == null ) {
noIndexReason = TextParser.supports(response.url(), response.getMimeType());
}
// check X-YACY-Index-Control
// With the X-YACY-Index-Control header set to "no-index" a client could disallow
// yacy to index the response returned as answer to a request
if ( noIndexReason == null && response.requestProhibitsIndexing() ) {
noIndexReason = "X-YACY-Index-Control header prohibits indexing";
}
// check accepted domain / localhost accesses
if ( noIndexReason == null ) {
noIndexReason = this.crawlStacker.urlInAcceptedDomain(response.url());
}
// in the noIndexReason is set, indexing is not allowed
if ( noIndexReason != null ) {
// log cause and close queue
//if (log.isFine()) log.logFine("deQueue: not indexed any word in URL " + response.url() + "; cause: " + noIndexReason);
- removed ZURL data structure; removed also the ZURL data file - replaced load failure logging by information which is stored in Solr - fixed a bug with crawling of feeds: added must-match pattern application to feed urls to filter out such urls which shall not be in a wanted domain - delegatedURLs, which also used ZURLs are now temporary objects in memory
2013-09-17 15:27:02 +02:00
// create a new errorURL DB entry
this.crawlQueues.errorURL.push(response.url(), response.profile(), FailCategory.FINAL_PROCESS_CONTEXT, noIndexReason, -1);
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
// finish this entry
return "not allowed: " + noIndexReason;
}
- reduction of the concurrently running processes to make YaCy more adjusted to smaller and 1-core devices. - the workflow processor now starts no process at all. these are started as soon as parser/condenser/indexing queues are filled. - better abstraction
2013-04-25 11:33:17 +02:00
this.indexingDocumentProcessor.enQueue(new IndexingQueueEntry(
response,
null,
null));
return null;
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
}
public boolean processSurrogate(final String s) {
final File infile = new File(this.surrogatesInPath, s);
if ( !infile.exists() || !infile.canWrite() || !infile.canRead() ) {
return false;
}
final File outfile = new File(this.surrogatesOutPath, s);
//if (outfile.exists()) return false;
boolean moved = false;
if ( s.endsWith("xml.zip") ) {
// open the zip file with all the xml files in it
ZipInputStream zis = null;
try {
final InputStream is = new BufferedInputStream(new FileInputStream(infile));
zis = new ZipInputStream(is);
ZipEntry entry;
while ( (entry = zis.getNextEntry()) != null ) {
int size;
final byte[] buffer = new byte[2048];
final ByteArrayOutputStream baos = new ByteArrayOutputStream();
while ( (size = zis.read(buffer, 0, buffer.length)) != -1 ) {
baos.write(buffer, 0, size);
}
baos.flush();
processSurrogate(new ByteArrayInputStream(baos.toByteArray()), entry.getName());
baos.close();
if (shallTerminate()) break;
}
Added 'final' for all exception blocks as this helps the Java compiler to optimize memory usage Conflicts: source/net/yacy/search/Switchboard.java
2013-07-17 18:31:30 +02:00
} catch (final IOException e ) {
- refactoring of log to ConcurrentLog: jdk-based logger tend to block at java.util.logging.Logger.log(Logger.java:476) in concurrent environments. This makes logging a main performance issue. To overcome this problem, this is a add-on to jdk logging to put log entries on a concurrent message queue and log the messages one by one using a separate process. - FTPClient uses the concurrent logging instead of the log4j logger
2013-07-09 14:28:25 +02:00
ConcurrentLog.logException(e);
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
} finally {
moved = infile.renameTo(outfile);
Added 'final' for all exception blocks as this helps the Java compiler to optimize memory usage Conflicts: source/net/yacy/search/Switchboard.java
2013-07-17 18:31:30 +02:00
if (zis != null) try {zis.close();} catch (final IOException e) {}
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
}
return moved;
}
try {
InputStream is = new BufferedInputStream(new FileInputStream(infile));
if ( s.endsWith(".gz") ) {
is = new GZIPInputStream(is);
}
processSurrogate(is, infile.getName());
Added 'final' for all exception blocks as this helps the Java compiler to optimize memory usage Conflicts: source/net/yacy/search/Switchboard.java
2013-07-17 18:31:30 +02:00
} catch (final IOException e ) {
- refactoring of log to ConcurrentLog: jdk-based logger tend to block at java.util.logging.Logger.log(Logger.java:476) in concurrent environments. This makes logging a main performance issue. To overcome this problem, this is a add-on to jdk logging to put log entries on a concurrent message queue and log the messages one by one using a separate process. - FTPClient uses the concurrent logging instead of the log4j logger
2013-07-09 14:28:25 +02:00
ConcurrentLog.logException(e);
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
} finally {
if (!shallTerminate()) {
moved = infile.renameTo(outfile);
if ( moved ) {
// check if this file is already compressed, if not, compress now
if ( !outfile.getName().endsWith(".gz") ) {
final String gzname = outfile.getName() + ".gz";
final File gzfile = new File(outfile.getParentFile(), gzname);
try {
final OutputStream os =
new BufferedOutputStream(new GZIPOutputStream(new FileOutputStream(gzfile)));
FileUtils.copy(new BufferedInputStream(new FileInputStream(outfile)), os);
os.close();
if ( gzfile.exists() ) {
FileUtils.deletedelete(outfile);
}
Added 'final' for all exception blocks as this helps the Java compiler to optimize memory usage Conflicts: source/net/yacy/search/Switchboard.java
2013-07-17 18:31:30 +02:00
} catch (final FileNotFoundException e ) {
- refactoring of log to ConcurrentLog: jdk-based logger tend to block at java.util.logging.Logger.log(Logger.java:476) in concurrent environments. This makes logging a main performance issue. To overcome this problem, this is a add-on to jdk logging to put log entries on a concurrent message queue and log the messages one by one using a separate process. - FTPClient uses the concurrent logging instead of the log4j logger
2013-07-09 14:28:25 +02:00
ConcurrentLog.logException(e);
Added 'final' for all exception blocks as this helps the Java compiler to optimize memory usage Conflicts: source/net/yacy/search/Switchboard.java
2013-07-17 18:31:30 +02:00
} catch (final IOException e ) {
- refactoring of log to ConcurrentLog: jdk-based logger tend to block at java.util.logging.Logger.log(Logger.java:476) in concurrent environments. This makes logging a main performance issue. To overcome this problem, this is a add-on to jdk logging to put log entries on a concurrent message queue and log the messages one by one using a separate process. - FTPClient uses the concurrent logging instead of the log4j logger
2013-07-09 14:28:25 +02:00
ConcurrentLog.logException(e);
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
}
}
}
}
}
return moved;
}
public void processSurrogate(final InputStream is, final String name) throws IOException {
final SurrogateReader reader = new SurrogateReader(is, 100);
final Thread readerThread = new Thread(reader, name);
readerThread.start();
DCEntry surrogate;
Response response;
while ( (surrogate = reader.take()) != DCEntry.poison ) {
// check if url is in accepted domain
assert surrogate != null;
assert this.crawlStacker != null;
final String urlRejectReason =
this.crawlStacker.urlInAcceptedDomain(surrogate.getIdentifier(true));
if ( urlRejectReason != null ) {
- refactoring of log to ConcurrentLog: jdk-based logger tend to block at java.util.logging.Logger.log(Logger.java:476) in concurrent environments. This makes logging a main performance issue. To overcome this problem, this is a add-on to jdk logging to put log entries on a concurrent message queue and log the messages one by one using a separate process. - FTPClient uses the concurrent logging instead of the log4j logger
2013-07-09 14:28:25 +02:00
this.log.warn("Rejected URL '"
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
+ surrogate.getIdentifier(true)
+ "': "
+ urlRejectReason);
continue;
}
// create a queue entry
final Document document = surrogate.document();
final Request request =
new Request(
ASCII.getBytes(this.peers.mySeed().hash),
surrogate.getIdentifier(true),
null,
"",
surrogate.getDate(),
this.crawler.defaultSurrogateProfile.handle(),
0,
0,
0,
0);
response = new Response(request, null, null, this.crawler.defaultSurrogateProfile, false);
final IndexingQueueEntry queueEntry =
new IndexingQueueEntry(response, new Document[] {document}, null);
- reduction of the concurrently running processes to make YaCy more adjusted to smaller and 1-core devices. - the workflow processor now starts no process at all. these are started as soon as parser/condenser/indexing queues are filled. - better abstraction
2013-04-25 11:33:17 +02:00
this.indexingCondensementProcessor.enQueue(queueEntry);
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
if (shallTerminate()) break;
}
}
public int surrogateQueueSize() {
// count surrogates
final String[] surrogatelist = this.surrogatesInPath.list();
if ( surrogatelist.length > 100 ) {
return 100;
}
int count = 0;
for ( final String s : surrogatelist ) {
if ( s.endsWith(".xml") ) {
count++;
}
if ( count >= 100 ) {
break;
}
}
return count;
}
public void surrogateFreeMem() {
// do nothing
}
public boolean surrogateProcess() {
// work off fresh entries from the proxy or from the crawler
final String cautionCause = onlineCaution();
if ( cautionCause != null ) {
if ( this.log.isFine() ) {
- refactoring of log to ConcurrentLog: jdk-based logger tend to block at java.util.logging.Logger.log(Logger.java:476) in concurrent environments. This makes logging a main performance issue. To overcome this problem, this is a add-on to jdk logging to put log entries on a concurrent message queue and log the messages one by one using a separate process. - FTPClient uses the concurrent logging instead of the log4j logger
2013-07-09 14:28:25 +02:00
this.log.fine("deQueue: online caution for "
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
+ cautionCause
+ ", omitting resource stack processing");
}
return false;
}
try {
// check surrogates
final String[] surrogatelist = this.surrogatesInPath.list();
fix for NPE if surrogates do not exist
2013-03-10 19:46:06 +01:00
if ( surrogatelist != null && surrogatelist.length > 0 ) {
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
// look if the is any xml inside
for ( final String surrogate : surrogatelist ) {
// check for interruption
checkInterruption();
if ( surrogate.endsWith(".xml")
|| surrogate.endsWith(".xml.gz")
|| surrogate.endsWith(".xml.zip") ) {
// read the surrogate file and store entry in index
if ( processSurrogate(surrogate) ) {
return true;
}
}
}
}
Added 'final' for all exception blocks as this helps the Java compiler to optimize memory usage Conflicts: source/net/yacy/search/Switchboard.java
2013-07-17 18:31:30 +02:00
} catch (final InterruptedException e ) {
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
return false;
}
return false;
}
made index storage from DHT search result concurrently. This prevents blocking by high CPU usage during search. Also: removed query from Solr for DHT search results; results are taken from the pending queue.
2013-03-02 10:25:52 +01:00
public void searchresultFreeMem() {
// do nothing
}
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
public int cleanupJobSize() {
int c = 1; // "es gibt immer was zu tun"
- removed ZURL data structure; removed also the ZURL data file - replaced load failure logging by information which is stored in Solr - fixed a bug with crawling of feeds: added must-match pattern application to feed urls to filter out such urls which shall not be in a wanted domain - delegatedURLs, which also used ZURLs are now temporary objects in memory
2013-09-17 15:27:02 +02:00
if ( (this.crawlQueues.delegatedURL.size() > 1000) ) {
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
c++;
}
if ( (this.crawlQueues.errorURL.stackSize() > 1000) ) {
c++;
}
for ( final EventOrigin origin : EventOrigin.values() ) {
if ( ResultURLs.getStackSize(origin) > 1000 ) {
c++;
}
}
return c;
}
fixed and enhanced postprocessing
2013-11-16 08:23:21 +01:00
public static boolean postprocessingRunning = false;
// if started, the following values are assigned for [collection1, webgraph]:
public static long[] postprocessingStartTime = new long[]{0,0}; // the start time for the processing; not started = 0
public static int[] postprocessingCount = new int[]{0,0}; // number of documents to be processed
added a postprocessing field into api/status_p.xml to show if the postprocessing task is running at that time (status: busy) or not (status:idle)
2013-07-12 14:29:22 +02:00
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
public boolean cleanupJob() {
fix for possible memory leaks
2013-03-13 17:55:37 +01:00
try {
// flush caches in used libraries
getting the trash out
2013-05-29 16:09:05 +02:00
pdfParser.clean_up_idiotic_PDFParser_font_cache_which_eats_up_tons_of_megabytes(); // eats up megabytes, see http://markmail.org/thread/quk5odee4hbsauhu
fix for possible memory leaks
2013-03-13 17:55:37 +01:00
// clear caches
enhanced did-you-mean (a bit): can now remember previously searched words (plus small enhancements)
2013-03-17 10:52:31 +01:00
if (WordCache.sizeCommonWords() > 1000) WordCache.clearCommonWords();
getting the trash out
2013-05-29 16:09:05 +02:00
Word.clearCache();
purge a lucene cache - possible memory leak fix
2013-11-18 22:47:35 +01:00
// Domains.clear();
FieldCache.DEFAULT.purgeAllCaches();
fix for possible memory leaks
2013-03-13 17:55:37 +01:00
// clean up image stack
ResultImages.clearQueues();
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
// flush the document compressor cache
Cache.commit();
Digest.cleanup(); // don't let caches become permanent memory leaks
// clear caches if necessary
fix for possible memory leaks
2013-03-13 17:55:37 +01:00
if ( !MemoryControl.request(128000000L, false) ) {
found and fixed a huge memory leak in solr caching (inside Solr). The not-flushed Solr cache is now handled in this way: - it is smaller by default - an Solr-internal process is started to flush the cache periodically (this does NOT clean the cache, just removes old objects) - a Solr-external process (the standard YaCy cleanup-process) now has direct access to the solr internal cache and flushes them completely. The time frame for such a flush is defined by the cleanup-process frequency, by default 10 minutes.
2013-11-07 10:01:44 +01:00
this.index.clearCaches();
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
SearchEventCache.cleanupEvents(false);
this.trail.clear();
Automatically flush the log cache if a short memory status is reached. For the default of 200 lines this can flush about 10MB.
2013-10-24 17:39:50 +02:00
GuiHandler.clear();
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
}
// set a random password if no password is configured
refactoring (usage of constant names for attributes of authentication check)
2014-01-05 04:23:44 +01:00
if ( getConfigBool(SwitchboardConstants.ADMIN_ACCOUNT_FOR_LOCALHOST, false)
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
&& getConfig(SwitchboardConstants.ADMIN_ACCOUNT_B64MD5, "").isEmpty() ) {
// make a 'random' password
setConfig(SwitchboardConstants.ADMIN_ACCOUNT_B64MD5, "0000" + this.genRandomPassword());
refactoring (usage of constant names for attributes of authentication check)
2014-01-05 04:23:44 +01:00
setConfig(SwitchboardConstants.ADMIN_ACCOUNT, "");
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
}
added a 'greedy learning' mechanismn which will cause that a 'fresh' yacy will load linked web pages from search results until the total number of web pages reaches 15000. This shall give fresh peers a 'boost' to get faster a personalized search index.
2013-06-11 14:42:30 +02:00
// stop greedylearning if limit is reached
if (getConfigBool(SwitchboardConstants.GREEDYLEARNING_ACTIVE, false)) {
long cs = this.index.fulltext().collectionSize();
if (cs > getConfigInt(SwitchboardConstants.GREEDYLEARNING_LIMIT_DOCCOUNT, 0)) {
setConfig(SwitchboardConstants.GREEDYLEARNING_ACTIVE, false);
- refactoring of log to ConcurrentLog: jdk-based logger tend to block at java.util.logging.Logger.log(Logger.java:476) in concurrent environments. This makes logging a main performance issue. To overcome this problem, this is a add-on to jdk logging to put log entries on a concurrent message queue and log the messages one by one using a separate process. - FTPClient uses the concurrent logging instead of the log4j logger
2013-07-09 14:28:25 +02:00
log.info("finishing greedy learning phase, size=" +cs);
added a 'greedy learning' mechanismn which will cause that a 'fresh' yacy will load linked web pages from search results until the total number of web pages reaches 15000. This shall give fresh peers a 'boost' to get faster a personalized search index.
2013-06-11 14:42:30 +02:00
}
}
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
// refresh recrawl dates
try {
CrawlProfile selentry;
for ( final byte[] handle : this.crawler.getActive() ) {
selentry = this.crawler.getActive(handle);
assert selentry.handle() != null : "profile.name = " + selentry.collectionName();
if ( selentry.handle() == null ) {
this.crawler.removeActive(handle);
continue;
}
boolean insert = false;
if ( selentry.name().equals(CrawlSwitchboard.CRAWL_PROFILE_PROXY) ) {
selentry.put(CrawlProfile.RECRAWL_IF_OLDER, Long.toString(CrawlProfile.getRecrawlDate(CrawlSwitchboard.CRAWL_PROFILE_PROXY_RECRAWL_CYCLE)));
insert = true;
}
if ( selentry.name().equals(CrawlSwitchboard.CRAWL_PROFILE_SNIPPET_LOCAL_TEXT) ) {
selentry.put(CrawlProfile.RECRAWL_IF_OLDER, Long.toString(CrawlProfile.getRecrawlDate(CrawlSwitchboard.CRAWL_PROFILE_SNIPPET_LOCAL_TEXT_RECRAWL_CYCLE)));
insert = true;
}
if ( selentry.name().equals(CrawlSwitchboard.CRAWL_PROFILE_SNIPPET_GLOBAL_TEXT) ) {
selentry.put(CrawlProfile.RECRAWL_IF_OLDER, Long.toString(CrawlProfile.getRecrawlDate(CrawlSwitchboard.CRAWL_PROFILE_SNIPPET_GLOBAL_TEXT_RECRAWL_CYCLE)));
insert = true;
}
prepartion for greedy crawl profiles and refactoring
2013-07-01 13:10:09 +02:00
if ( selentry.name().equals(CrawlSwitchboard.CRAWL_PROFILE_GREEDY_LEARNING_TEXT) ) {
selentry.put(CrawlProfile.RECRAWL_IF_OLDER, Long.toString(CrawlProfile.getRecrawlDate(CrawlSwitchboard.CRAWL_PROFILE_GREEDY_LEARNING_TEXT_RECRAWL_CYCLE)));
insert = true;
}
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
if ( selentry.name().equals(CrawlSwitchboard.CRAWL_PROFILE_SNIPPET_LOCAL_MEDIA) ) {
selentry.put(CrawlProfile.RECRAWL_IF_OLDER, Long.toString(CrawlProfile.getRecrawlDate(CrawlSwitchboard.CRAWL_PROFILE_SNIPPET_LOCAL_MEDIA_RECRAWL_CYCLE)));
insert = true;
}
if ( selentry.name().equals(CrawlSwitchboard.CRAWL_PROFILE_SNIPPET_GLOBAL_MEDIA) ) {
selentry.put(CrawlProfile.RECRAWL_IF_OLDER, Long.toString(CrawlProfile.getRecrawlDate(CrawlSwitchboard.CRAWL_PROFILE_SNIPPET_GLOBAL_MEDIA_RECRAWL_CYCLE)));
insert = true;
}
if ( selentry.name().equals(CrawlSwitchboard.CRAWL_PROFILE_SURROGATE) ) {
selentry.put(CrawlProfile.RECRAWL_IF_OLDER, Long.toString(CrawlProfile.getRecrawlDate(CrawlSwitchboard.CRAWL_PROFILE_SURROGATE_RECRAWL_CYCLE)));
insert = true;
}
if ( insert ) {
this.crawler.putActive(UTF8.getBytes(selentry.handle()), selentry);
}
}
Added 'final' for all exception blocks as this helps the Java compiler to optimize memory usage Conflicts: source/net/yacy/search/Switchboard.java
2013-07-17 18:31:30 +02:00
} catch (final Exception e ) {
- refactoring of log to ConcurrentLog: jdk-based logger tend to block at java.util.logging.Logger.log(Logger.java:476) in concurrent environments. This makes logging a main performance issue. To overcome this problem, this is a add-on to jdk logging to put log entries on a concurrent message queue and log the messages one by one using a separate process. - FTPClient uses the concurrent logging instead of the log4j logger
2013-07-09 14:28:25 +02:00
ConcurrentLog.logException(e);
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
}
// close unused connections
ConnectionInfo.cleanUp();
// clean up delegated stack
checkInterruption();
- removed ZURL data structure; removed also the ZURL data file - replaced load failure logging by information which is stored in Solr - fixed a bug with crawling of feeds: added must-match pattern application to feed urls to filter out such urls which shall not be in a wanted domain - delegatedURLs, which also used ZURLs are now temporary objects in memory
2013-09-17 15:27:02 +02:00
if ( (this.crawlQueues.delegatedURL.size() > 1000) ) {
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
if ( this.log.isFine() ) {
- refactoring of log to ConcurrentLog: jdk-based logger tend to block at java.util.logging.Logger.log(Logger.java:476) in concurrent environments. This makes logging a main performance issue. To overcome this problem, this is a add-on to jdk logging to put log entries on a concurrent message queue and log the messages one by one using a separate process. - FTPClient uses the concurrent logging instead of the log4j logger
2013-07-09 14:28:25 +02:00
this.log.fine("Cleaning Delegated-URLs report stack, "
- removed ZURL data structure; removed also the ZURL data file - replaced load failure logging by information which is stored in Solr - fixed a bug with crawling of feeds: added must-match pattern application to feed urls to filter out such urls which shall not be in a wanted domain - delegatedURLs, which also used ZURLs are now temporary objects in memory
2013-09-17 15:27:02 +02:00
+ this.crawlQueues.delegatedURL.size()
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
+ " entries on stack");
}
- removed ZURL data structure; removed also the ZURL data file - replaced load failure logging by information which is stored in Solr - fixed a bug with crawling of feeds: added must-match pattern application to feed urls to filter out such urls which shall not be in a wanted domain - delegatedURLs, which also used ZURLs are now temporary objects in memory
2013-09-17 15:27:02 +02:00
this.crawlQueues.delegatedURL.clear();
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
}
// clean up error stack
checkInterruption();
if ( (this.crawlQueues.errorURL.stackSize() > 1000) ) {
if ( this.log.isFine() ) {
- refactoring of log to ConcurrentLog: jdk-based logger tend to block at java.util.logging.Logger.log(Logger.java:476) in concurrent environments. This makes logging a main performance issue. To overcome this problem, this is a add-on to jdk logging to put log entries on a concurrent message queue and log the messages one by one using a separate process. - FTPClient uses the concurrent logging instead of the log4j logger
2013-07-09 14:28:25 +02:00
this.log.fine("Cleaning Error-URLs report stack, "
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
+ this.crawlQueues.errorURL.stackSize()
+ " entries on stack");
}
this.crawlQueues.errorURL.clearStack();
}
// clean up loadedURL stack
for ( final EventOrigin origin : EventOrigin.values() ) {
checkInterruption();
if ( ResultURLs.getStackSize(origin) > 1000 ) {
if ( this.log.isFine() ) {
- refactoring of log to ConcurrentLog: jdk-based logger tend to block at java.util.logging.Logger.log(Logger.java:476) in concurrent environments. This makes logging a main performance issue. To overcome this problem, this is a add-on to jdk logging to put log entries on a concurrent message queue and log the messages one by one using a separate process. - FTPClient uses the concurrent logging instead of the log4j logger
2013-07-09 14:28:25 +02:00
this.log.fine("Cleaning Loaded-URLs report stack, "
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
+ ResultURLs.getStackSize(origin)
+ " entries on stack "
+ origin.getCode());
}
ResultURLs.clearStack(origin);
}
}
// clean up news
checkInterruption();
try {
if ( this.log.isFine() ) {
- refactoring of log to ConcurrentLog: jdk-based logger tend to block at java.util.logging.Logger.log(Logger.java:476) in concurrent environments. This makes logging a main performance issue. To overcome this problem, this is a add-on to jdk logging to put log entries on a concurrent message queue and log the messages one by one using a separate process. - FTPClient uses the concurrent logging instead of the log4j logger
2013-07-09 14:28:25 +02:00
this.log.fine("Cleaning Incoming News, "
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
+ this.peers.newsPool.size(NewsPool.INCOMING_DB)
+ " entries on stack");
}
this.peers.newsPool.automaticProcess(this.peers);
Added 'final' for all exception blocks as this helps the Java compiler to optimize memory usage Conflicts: source/net/yacy/search/Switchboard.java
2013-07-17 18:31:30 +02:00
} catch (final Exception e ) {
- refactoring of log to ConcurrentLog: jdk-based logger tend to block at java.util.logging.Logger.log(Logger.java:476) in concurrent environments. This makes logging a main performance issue. To overcome this problem, this is a add-on to jdk logging to put log entries on a concurrent message queue and log the messages one by one using a separate process. - FTPClient uses the concurrent logging instead of the log4j logger
2013-07-09 14:28:25 +02:00
ConcurrentLog.logException(e);
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
}
if ( getConfigBool("cleanup.deletionProcessedNews", true) ) {
this.peers.newsPool.clear(NewsPool.PROCESSED_DB);
}
if ( getConfigBool("cleanup.deletionPublishedNews", true) ) {
this.peers.newsPool.clear(NewsPool.PUBLISHED_DB);
}
// clean up seed-dbs
if ( getConfigBool("routing.deleteOldSeeds.permission", true) ) {
final long deleteOldSeedsTime =
getConfigLong("routing.deleteOldSeeds.time", 7) * 24 * 3600000;
Iterator<Seed> e = this.peers.seedsSortedDisconnected(true, Seed.LASTSEEN);
Seed seed = null;
final List<String> deleteQueue = new ArrayList<String>();
checkInterruption();
// clean passive seeds
while ( e.hasNext() ) {
seed = e.next();
if ( seed != null ) {
//list is sorted -> break when peers are too young to delete
if ( !seed.isLastSeenTimeout(deleteOldSeedsTime) ) {
break;
}
deleteQueue.add(seed.hash);
}
}
for ( int i = 0; i < deleteQueue.size(); ++i ) {
this.peers.removeDisconnected(deleteQueue.get(i));
}
deleteQueue.clear();
e = this.peers.seedsSortedPotential(true, Seed.LASTSEEN);
checkInterruption();
// clean potential seeds
while ( e.hasNext() ) {
seed = e.next();
if ( seed != null ) {
//list is sorted -> break when peers are too young to delete
if ( !seed.isLastSeenTimeout(deleteOldSeedsTime) ) {
break;
}
deleteQueue.add(seed.hash);
}
}
for ( int i = 0; i < deleteQueue.size(); ++i ) {
this.peers.removePotential(deleteQueue.get(i));
}
}
// check if update is available and
// if auto-update is activated perform an automatic installation and restart
final yacyRelease updateVersion = yacyRelease.rulebasedUpdateInfo(false);
if ( updateVersion != null ) {
// there is a version that is more recent. Load it and re-start with it
- refactoring of log to ConcurrentLog: jdk-based logger tend to block at java.util.logging.Logger.log(Logger.java:476) in concurrent environments. This makes logging a main performance issue. To overcome this problem, this is a add-on to jdk logging to put log entries on a concurrent message queue and log the messages one by one using a separate process. - FTPClient uses the concurrent logging instead of the log4j logger
2013-07-09 14:28:25 +02:00
this.log.info("AUTO-UPDATE: downloading more recent release " + updateVersion.getUrl());
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
final File downloaded = updateVersion.downloadRelease();
final boolean devenvironment = new File(this.getAppPath(), ".git").exists();
if ( devenvironment ) {
this.log
- refactoring of log to ConcurrentLog: jdk-based logger tend to block at java.util.logging.Logger.log(Logger.java:476) in concurrent environments. This makes logging a main performance issue. To overcome this problem, this is a add-on to jdk logging to put log entries on a concurrent message queue and log the messages one by one using a separate process. - FTPClient uses the concurrent logging instead of the log4j logger
2013-07-09 14:28:25 +02:00
.info("AUTO-UPDATE: omitting update because this is a development environment");
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
} else if ( (downloaded == null) || (!downloaded.exists()) || (downloaded.length() == 0) ) {
this.log
- refactoring of log to ConcurrentLog: jdk-based logger tend to block at java.util.logging.Logger.log(Logger.java:476) in concurrent environments. This makes logging a main performance issue. To overcome this problem, this is a add-on to jdk logging to put log entries on a concurrent message queue and log the messages one by one using a separate process. - FTPClient uses the concurrent logging instead of the log4j logger
2013-07-09 14:28:25 +02:00
.info("AUTO-UPDATE: omitting update because download failed (file cannot be found, is too small or signature is bad)");
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
} else {
yacyRelease.deployRelease(downloaded);
terminate(10, "auto-update to install " + downloaded.getName());
- refactoring of log to ConcurrentLog: jdk-based logger tend to block at java.util.logging.Logger.log(Logger.java:476) in concurrent environments. This makes logging a main performance issue. To overcome this problem, this is a add-on to jdk logging to put log entries on a concurrent message queue and log the messages one by one using a separate process. - FTPClient uses the concurrent logging instead of the log4j logger
2013-07-09 14:28:25 +02:00
this.log.info("AUTO-UPDATE: deploy and restart initiated");
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
}
}
// initiate broadcast about peer startup to spread supporter url
if ( !isRobinsonMode() && this.peers.newsPool.size(NewsPool.OUTGOING_DB) == 0 ) {
// read profile
final Properties profile = new Properties();
FileInputStream fileIn = null;
try {
fileIn = new FileInputStream(new File("DATA/SETTINGS/profile.txt"));
profile.load(fileIn);
Added 'final' for all exception blocks as this helps the Java compiler to optimize memory usage Conflicts: source/net/yacy/search/Switchboard.java
2013-07-17 18:31:30 +02:00
} catch (final IOException e ) {
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
} finally {
if ( fileIn != null ) {
try {
fileIn.close();
Added 'final' for all exception blocks as this helps the Java compiler to optimize memory usage Conflicts: source/net/yacy/search/Switchboard.java
2013-07-17 18:31:30 +02:00
} catch (final Exception e ) {
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
}
}
}
final String homepage = (String) profile.get("homepage");
if ( (homepage != null) && (homepage.length() > 10) ) {
final Properties news = new Properties();
news.put("homepage", profile.get("homepage"));
this.peers.newsPool.publishMyNews(
this.peers.mySeed(),
NewsPool.CATEGORY_PROFILE_BROADCAST,
news);
}
}
// update the cluster set
this.clusterhashes = this.peers.clusterHashes(getConfig("cluster.peers.yacydomain", ""));
// check if we are reachable and try to map port again if not (e.g. when router rebooted)
if ( getConfigBool(SwitchboardConstants.UPNP_ENABLED, false) && this.peers.mySeed().isJunior() ) {
UPnP.addPortMapping();
}
// after all clean up is done, check the resource usage
this.observer.resourceObserverJob();
// cleanup cached search failures
if ( getConfigBool(SwitchboardConstants.NETWORK_SEARCHVERIFY, false)
&& this.peers.mySeed().getFlagAcceptRemoteIndex() ) {
this.tables.cleanFailURLS(getConfigLong("cleanup.failedSearchURLtimeout", -1));
}
// periodically store the triple store
if (getConfigBool("triplestore.persistent", false)) {
JenaTripleStore.saveAll();
}
enhanced postprocessing: fixed bugs, enable proper postprocessing also without the harvestingkey, remove crawl profiles after postprocessing, speed-up for clickdepth computation.
2013-10-16 11:27:06 +02:00
// clean up profiles
checkInterruption();
added the clickdepth post-processing: some links may have 'shortcuts' to already calculated click depths. There are then calculated if the crawl buffer is empty and therefore no new 'shortcuts' can be discovered. The status of the clickdepth stack (to-be-processed) can be seen using a solr search command like this: http://localhost:8090/solr/select?q=process_sxt:[*%20TO%20*]&start=0&rows=30&fl=sku,clickdepth_i,process_sxt
2013-01-04 16:37:39 +01:00
// if no crawl is running and processing is activated:
// execute the (post-) processing steps for all entries that have a process tag assigned
fixed and enhanced postprocessing
2013-11-16 08:23:21 +01:00
Fulltext fulltext = index.fulltext();
CollectionConfiguration collection1Configuration = fulltext.getDefaultConfiguration();
WebgraphConfiguration webgraphConfiguration = fulltext.getWebgraphConfiguration();
enhanced postprocessing: fixed bugs, enable proper postprocessing also without the harvestingkey, remove crawl profiles after postprocessing, speed-up for clickdepth computation.
2013-10-16 11:27:06 +02:00
if (!this.crawlJobIsPaused(SwitchboardConstants.CRAWLJOB_LOCAL_CRAWL)) {
fixed and enhanced postprocessing
2013-11-16 08:23:21 +01:00
// we optimize first because that is useful for postprocessing
do a light optimization at the end of a crawl postprocessing
2013-07-13 19:09:46 +02:00
int proccount = 0;
fixed and enhanced postprocessing
2013-11-16 08:23:21 +01:00
ReferenceReportCache rrCache = index.getReferenceReportCache();
ClickdepthCache clickdepthCache = index.getClickdepthCache(rrCache);
Set<String> deletionCandidates = collection1Configuration.contains(CollectionSchema.harvestkey_s.getSolrFieldName()) ?
this.crawler.getFinishesProfiles(this.crawlQueues) : new HashSet<String>();
int cleanupByHarvestkey = deletionCandidates.size();
enhanced webgraph processing
2013-12-04 01:54:45 +01:00
boolean processCollection = collection1Configuration.contains(CollectionSchema.process_sxt) && (index.connectedCitation() || fulltext.useWebgraph());
boolean processWebgraph = webgraphConfiguration.contains(WebgraphSchema.process_sxt) && fulltext.useWebgraph();
removed optimization bevore postprocessing because that may cause a time-out which will cause that postprocessing fails.
2013-12-04 16:04:29 +01:00
boolean allCrawlsFinished = this.crawler.allCrawlsFinished(this.crawlQueues);
fixed and enhanced postprocessing
2013-11-16 08:23:21 +01:00
if ((processCollection || processWebgraph) && (cleanupByHarvestkey > 0 || allCrawlsFinished)) {
if (cleanupByHarvestkey > 0) {
// run postprocessing on these profiles
postprocessingRunning = true;
postprocessingStartTime[0] = System.currentTimeMillis();
try {postprocessingCount[0] = (int) fulltext.getDefaultConnector().getCountByQuery(CollectionSchema.process_sxt.getSolrFieldName() + ":[* TO *]");} catch (IOException e) {}
for (String profileHash: deletionCandidates) proccount += collection1Configuration.postprocessing(index, rrCache, clickdepthCache, profileHash);
postprocessingStartTime[0] = 0;
try {postprocessingCount[0] = (int) fulltext.getDefaultConnector().getCountByQuery(CollectionSchema.process_sxt.getSolrFieldName() + ":[* TO *]");} catch (IOException e) {} // should be zero but you never know
if (processWebgraph) {
postprocessingStartTime[1] = System.currentTimeMillis();
try {postprocessingCount[1] = (int) fulltext.getWebgraphConnector().getCountByQuery(WebgraphSchema.process_sxt.getSolrFieldName() + ":[* TO *]");} catch (IOException e) {}
for (String profileHash: deletionCandidates) proccount += webgraphConfiguration.postprocessing(index, clickdepthCache, profileHash);
postprocessingStartTime[1] = 0;
try {postprocessingCount[1] = (int) fulltext.getWebgraphConnector().getCountByQuery(WebgraphSchema.process_sxt.getSolrFieldName() + ":[* TO *]");} catch (IOException e) {}
}
this.crawler.cleanProfiles(deletionCandidates);
log.info("cleanup removed " + cleanupByHarvestkey + " crawl profiles, post-processed " + proccount + " documents");
} else if (allCrawlsFinished) {
// run postprocessing on all profiles
postprocessingRunning = true;
postprocessingStartTime[0] = System.currentTimeMillis();
try {postprocessingCount[0] = (int) fulltext.getDefaultConnector().getCountByQuery(CollectionSchema.process_sxt.getSolrFieldName() + ":[* TO *]");} catch (IOException e) {}
proccount += collection1Configuration.postprocessing(index, rrCache, clickdepthCache, null);
postprocessingStartTime[0] = 0;
try {postprocessingCount[0] = (int) fulltext.getDefaultConnector().getCountByQuery(CollectionSchema.process_sxt.getSolrFieldName() + ":[* TO *]");} catch (IOException e) {} // should be zero but you never know
if (processWebgraph) {
postprocessingStartTime[1] = System.currentTimeMillis();
try {postprocessingCount[1] = (int) fulltext.getWebgraphConnector().getCountByQuery(WebgraphSchema.process_sxt.getSolrFieldName() + ":[* TO *]");} catch (IOException e) {}
proccount += webgraphConfiguration.postprocessing(index, clickdepthCache, null);
postprocessingStartTime[1] = 0;
try {postprocessingCount[1] = (int) fulltext.getWebgraphConnector().getCountByQuery(WebgraphSchema.process_sxt.getSolrFieldName() + ":[* TO *]");} catch (IOException e) {}
}
this.crawler.cleanProfiles(this.crawler.getActiveProfiles());
log.info("cleanup post-processed " + proccount + " documents");
}
}
more monitoring for postprocessing and enhanced layout in Crawler monitor page
2013-11-16 18:23:14 +01:00
this.index.fulltext().commit(true); // without a commit the success is not visible in the monitoring
removed optimization bevore postprocessing because that may cause a time-out which will cause that postprocessing fails.
2013-12-04 16:04:29 +01:00
if (allCrawlsFinished) {
postprocessingRunning = true;
// flush caches
Domains.clear();
this.crawlQueues.noticeURL.clear();
// do solr optimization
long idleSearch = System.currentTimeMillis() - this.localSearchLastAccess;
long idleAdmin = System.currentTimeMillis() - this.adminAuthenticationLastAccess;
long deltaOptimize = System.currentTimeMillis() - this.optimizeLastRun;
boolean optimizeRequired = deltaOptimize > 60000 * 60 * 3; // 3 hours
int opts = Math.max(1, (int) (fulltext.collectionSize() / 5000000));
log.info("Solr auto-optimization: idleSearch=" + idleSearch + ", idleAdmin=" + idleAdmin + ", deltaOptimize=" + deltaOptimize + ", proccount=" + proccount);
if (idleAdmin > 600000) {
// only run optimization if the admin is idle (10 minutes)
if (proccount > 0) {
opts++; // have postprocessings will force optimazion with one more Segment which is small an quick
optimizeRequired = true;
}
if (optimizeRequired) {
if (idleSearch < 600000) opts++; // < 10 minutes idle time will cause a optimization with one more Segment which is small an quick
log.info("Solr auto-optimization: running solr.optimize(" + opts + ")");
fulltext.optimize(opts);
this.optimizeLastRun = System.currentTimeMillis();
}
}
}
more monitoring for postprocessing and enhanced layout in Crawler monitor page
2013-11-16 18:23:14 +01:00
postprocessingStartTime = new long[]{0,0}; // the start time for the processing; not started = 0
added a postprocessing field into api/status_p.xml to show if the postprocessing task is running at that time (status: busy) or not (status:idle)
2013-07-12 14:29:22 +02:00
postprocessingRunning = false;
added the clickdepth post-processing: some links may have 'shortcuts' to already calculated click depths. There are then calculated if the crawl buffer is empty and therefore no new 'shortcuts' can be discovered. The status of the clickdepth stack (to-be-processed) can be seen using a solr search command like this: http://localhost:8090/solr/select?q=process_sxt:[*%20TO%20*]&start=0&rows=30&fl=sku,clickdepth_i,process_sxt
2013-01-04 16:37:39 +01:00
}
activate api actions after postprocessing of crawls
2013-07-12 16:05:48 +02:00
// execute api actions; this must be done after postprocessing because
// these actions may also influence the search index/ call optimize steps
execAPIActions();
added the clickdepth post-processing: some links may have 'shortcuts' to already calculated click depths. There are then calculated if the crawl buffer is empty and therefore no new 'shortcuts' can be discovered. The status of the clickdepth stack (to-be-processed) can be seen using a solr search command like this: http://localhost:8090/solr/select?q=process_sxt:[*%20TO%20*]&start=0&rows=30&fl=sku,clickdepth_i,process_sxt
2013-01-04 16:37:39 +01:00
logging of deadlocks (if any) during cleanup process
2013-07-17 12:38:58 +02:00
// show deadlocks if there are any in the log
if (Memory.deadlocks() > 0) Memory.logDeadlocks();
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
return true;
Added 'final' for all exception blocks as this helps the Java compiler to optimize memory usage Conflicts: source/net/yacy/search/Switchboard.java
2013-07-17 18:31:30 +02:00
} catch (final InterruptedException e ) {
- refactoring of log to ConcurrentLog: jdk-based logger tend to block at java.util.logging.Logger.log(Logger.java:476) in concurrent environments. This makes logging a main performance issue. To overcome this problem, this is a add-on to jdk logging to put log entries on a concurrent message queue and log the messages one by one using a separate process. - FTPClient uses the concurrent logging instead of the log4j logger
2013-07-09 14:28:25 +02:00
this.log.info("cleanupJob: Shutdown detected");
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
return false;
}
}
private void execAPIActions() {
// execute scheduled API actions
Tables.Row row;
final Collection<String> pks = new LinkedHashSet<String>();
final Date now = new Date();
try {
final Iterator<Tables.Row> plainIterator = this.tables.iterator(WorkTables.TABLE_API_NAME);
final Iterator<Tables.Row> mapIterator = this.tables.orderBy(plainIterator, -1, WorkTables.TABLE_API_COL_DATE_RECORDING).iterator();
while (mapIterator.hasNext()) {
row = mapIterator.next();
if (row == null) continue;
// select api calls according to scheduler settings
final Date date_next_exec = row.get(WorkTables.TABLE_API_COL_DATE_NEXT_EXEC, (Date) null);
if (date_next_exec != null && now.after(date_next_exec)) pks.add(UTF8.String(row.getPK()));
// select api calls according to event settings
final String kind = row.get(WorkTables.TABLE_API_COL_APICALL_EVENT_KIND, "off");
if (!"off".equals(kind)) {
String action = row.get(WorkTables.TABLE_API_COL_APICALL_EVENT_ACTION, "startup");
if ("startup".equals(action)) {
if (startupAction) {
pks.add(UTF8.String(row.getPK()));
if ("once".equals(kind)) {
row.put(WorkTables.TABLE_API_COL_APICALL_EVENT_KIND, "off");
sb.tables.update(WorkTables.TABLE_API_NAME, row);
}
}
} else try {
SimpleDateFormat dateFormat = new SimpleDateFormat("yyyyMMddHHmm");
long d = dateFormat.parse(dateFormat.format(new Date()).substring(0, 8) + action).getTime();
long cycle = getThread(SwitchboardConstants.CLEANUP).getBusySleep();
if (d < System.currentTimeMillis() && System.currentTimeMillis() - d < cycle) {
pks.add(UTF8.String(row.getPK()));
if ("once".equals(kind)) {
row.put(WorkTables.TABLE_API_COL_APICALL_EVENT_KIND, "off");
row.put(WorkTables.TABLE_API_COL_DATE_NEXT_EXEC, "");
sb.tables.update(WorkTables.TABLE_API_NAME, row);
}
}
Added 'final' for all exception blocks as this helps the Java compiler to optimize memory usage Conflicts: source/net/yacy/search/Switchboard.java
2013-07-17 18:31:30 +02:00
} catch (final ParseException e) {}
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
}
}
} catch (final IOException e) {
- refactoring of log to ConcurrentLog: jdk-based logger tend to block at java.util.logging.Logger.log(Logger.java:476) in concurrent environments. This makes logging a main performance issue. To overcome this problem, this is a add-on to jdk logging to put log entries on a concurrent message queue and log the messages one by one using a separate process. - FTPClient uses the concurrent logging instead of the log4j logger
2013-07-09 14:28:25 +02:00
ConcurrentLog.logException(e);
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
}
for (final String pk : pks) {
try {
row = this.tables.select(WorkTables.TABLE_API_NAME, UTF8.getBytes(pk));
WorkTables.calculateAPIScheduler(row, true); // calculate next update time
this.tables.update(WorkTables.TABLE_API_NAME, row);
Added 'final' for all exception blocks as this helps the Java compiler to optimize memory usage Conflicts: source/net/yacy/search/Switchboard.java
2013-07-17 18:31:30 +02:00
} catch (final Throwable e ) {
- refactoring of log to ConcurrentLog: jdk-based logger tend to block at java.util.logging.Logger.log(Logger.java:476) in concurrent environments. This makes logging a main performance issue. To overcome this problem, this is a add-on to jdk logging to put log entries on a concurrent message queue and log the messages one by one using a separate process. - FTPClient uses the concurrent logging instead of the log4j logger
2013-07-09 14:28:25 +02:00
ConcurrentLog.logException(e);
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
continue;
}
}
startupAction = false;
// execute api calls
fixed shutdown authenticaton problem
2014-01-06 01:48:54 +01:00
final Map<String, Integer> callResult = this.tables.execAPICalls("localhost", (int) getConfigLong("port", 8090), pks, getConfig(SwitchboardConstants.ADMIN_ACCOUNT_B64MD5, ""));
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
for ( final Map.Entry<String, Integer> call : callResult.entrySet() ) {
- refactoring of log to ConcurrentLog: jdk-based logger tend to block at java.util.logging.Logger.log(Logger.java:476) in concurrent environments. This makes logging a main performance issue. To overcome this problem, this is a add-on to jdk logging to put log entries on a concurrent message queue and log the messages one by one using a separate process. - FTPClient uses the concurrent logging instead of the log4j logger
2013-07-09 14:28:25 +02:00
this.log.info("Scheduler executed api call, response " + call.getValue() + ": " + call.getKey());
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
}
}
/**
* With this function the crawling process can be paused
*
* @param jobType
*/
public void pauseCrawlJob(final String jobType, String cause) {
final Object[] status = this.crawlJobsStatus.get(jobType);
synchronized ( status[SwitchboardConstants.CRAWLJOB_SYNC] ) {
status[SwitchboardConstants.CRAWLJOB_STATUS] = Boolean.TRUE;
}
setConfig(jobType + "_isPaused", "true");
setConfig(jobType + "_isPaused_cause", cause);
- refactoring of log to ConcurrentLog: jdk-based logger tend to block at java.util.logging.Logger.log(Logger.java:476) in concurrent environments. This makes logging a main performance issue. To overcome this problem, this is a add-on to jdk logging to put log entries on a concurrent message queue and log the messages one by one using a separate process. - FTPClient uses the concurrent logging instead of the log4j logger
2013-07-09 14:28:25 +02:00
log.warn("Crawl job '" + jobType + "' is paused: " + cause);
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
}
/**
* Continue the previously paused crawling
*
* @param jobType
*/
public void continueCrawlJob(final String jobType) {
final Object[] status = this.crawlJobsStatus.get(jobType);
synchronized ( status[SwitchboardConstants.CRAWLJOB_SYNC] ) {
if ( ((Boolean) status[SwitchboardConstants.CRAWLJOB_STATUS]).booleanValue() ) {
status[SwitchboardConstants.CRAWLJOB_STATUS] = Boolean.FALSE;
status[SwitchboardConstants.CRAWLJOB_SYNC].notifyAll();
}
}
setConfig(jobType + "_isPaused", "false");
}
/**
* @param jobType
* @return <code>true</code> if crawling was paused or <code>false</code> otherwise
*/
public boolean crawlJobIsPaused(final String jobType) {
final Object[] status = this.crawlJobsStatus.get(jobType);
synchronized ( status[SwitchboardConstants.CRAWLJOB_SYNC] ) {
return ((Boolean) status[SwitchboardConstants.CRAWLJOB_STATUS]).booleanValue();
}
}
public IndexingQueueEntry parseDocument(final IndexingQueueEntry in) {
in.queueEntry.updateStatus(Response.QUEUE_STATE_PARSING);
Document[] documents = null;
try {
documents = parseDocument(in.queueEntry);
Added 'final' for all exception blocks as this helps the Java compiler to optimize memory usage Conflicts: source/net/yacy/search/Switchboard.java
2013-07-17 18:31:30 +02:00
} catch (final InterruptedException e ) {
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
documents = null;
Added 'final' for all exception blocks as this helps the Java compiler to optimize memory usage Conflicts: source/net/yacy/search/Switchboard.java
2013-07-17 18:31:30 +02:00
} catch (final Exception e ) {
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
documents = null;
}
if ( documents == null ) {
return null;
- removed ZURL data structure; removed also the ZURL data file - replaced load failure logging by information which is stored in Solr - fixed a bug with crawling of feeds: added must-match pattern application to feed urls to filter out such urls which shall not be in a wanted domain - delegatedURLs, which also used ZURLs are now temporary objects in memory
2013-09-17 15:27:02 +02:00
}
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
return new IndexingQueueEntry(in.queueEntry, documents, null);
}
private Document[] parseDocument(final Response response) throws InterruptedException {
Document[] documents = null;
added (partly commented) test code for url rewrite methods .. to be completed
2013-11-10 18:50:34 +01:00
//final Pattern rewritePattern = Pattern.compile(";jsessionid.*");
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
final EventOrigin processCase = response.processCase(this.peers.mySeed().hash);
if ( this.log.isFine() ) {
- refactoring of log to ConcurrentLog: jdk-based logger tend to block at java.util.logging.Logger.log(Logger.java:476) in concurrent environments. This makes logging a main performance issue. To overcome this problem, this is a add-on to jdk logging to put log entries on a concurrent message queue and log the messages one by one using a separate process. - FTPClient uses the concurrent logging instead of the log4j logger
2013-07-09 14:28:25 +02:00
this.log.fine(
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
"processResourceStack processCase=" + processCase
+ ", depth=" + response.depth()
+ ", maxDepth=" + ((response.profile() == null) ? "null" : Integer.toString(response.profile().depth()))
+ ", must-match=" + ((response.profile() == null) ? "null" : response.profile().urlMustMatchPattern().toString())
+ ", must-not-match=" + ((response.profile() == null) ? "null" : response.profile().urlMustNotMatchPattern().toString())
+ ", initiatorHash=" + ((response.initiator() == null) ? "null" : ASCII.String(response.initiator()))
+ ", url=" + response.url()); // DEBUG
}
// PARSE CONTENT
final long parsingStartTime = System.currentTimeMillis();
if ( response.getContent() == null ) {
// fetch the document from cache
response.setContent(Cache.getContent(response.url().hash()));
if ( response.getContent() == null ) {
- refactoring of log to ConcurrentLog: jdk-based logger tend to block at java.util.logging.Logger.log(Logger.java:476) in concurrent environments. This makes logging a main performance issue. To overcome this problem, this is a add-on to jdk logging to put log entries on a concurrent message queue and log the messages one by one using a separate process. - FTPClient uses the concurrent logging instead of the log4j logger
2013-07-09 14:28:25 +02:00
this.log.warn("the resource '" + response.url() + "' is missing in the cache.");
- removed ZURL data structure; removed also the ZURL data file - replaced load failure logging by information which is stored in Solr - fixed a bug with crawling of feeds: added must-match pattern application to feed urls to filter out such urls which shall not be in a wanted domain - delegatedURLs, which also used ZURLs are now temporary objects in memory
2013-09-17 15:27:02 +02:00
// create a new errorURL DB entry
this.crawlQueues.errorURL.push(response.url(), response.profile(), FailCategory.FINAL_LOAD_CONTEXT, "missing in cache", -1);
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
return null;
}
}
assert response.getContent() != null;
try {
// parse the document
documents =
TextParser.parseSource(
- replaced the properties object in AnchorURL with distinct variables for anchor attributes. - this caused that large portions of the parser code had to be adopted as well - added a counter target_order_i for anchor links in webgraph computation
2013-09-15 23:27:04 +02:00
new AnchorURL(response.url()),
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
response.getMimeType(),
response.getCharacterEncoding(),
response.getContent());
if ( documents == null ) {
throw new Parser.Failure("Parser returned null.", response.url());
}
Added 'final' for all exception blocks as this helps the Java compiler to optimize memory usage Conflicts: source/net/yacy/search/Switchboard.java
2013-07-17 18:31:30 +02:00
} catch (final Parser.Failure e ) {
- refactoring of log to ConcurrentLog: jdk-based logger tend to block at java.util.logging.Logger.log(Logger.java:476) in concurrent environments. This makes logging a main performance issue. To overcome this problem, this is a add-on to jdk logging to put log entries on a concurrent message queue and log the messages one by one using a separate process. - FTPClient uses the concurrent logging instead of the log4j logger
2013-07-09 14:28:25 +02:00
this.log.warn("Unable to parse the resource '" + response.url() + "'. " + e.getMessage());
- removed ZURL data structure; removed also the ZURL data file - replaced load failure logging by information which is stored in Solr - fixed a bug with crawling of feeds: added must-match pattern application to feed urls to filter out such urls which shall not be in a wanted domain - delegatedURLs, which also used ZURLs are now temporary objects in memory
2013-09-17 15:27:02 +02:00
// create a new errorURL DB entry
this.crawlQueues.errorURL.push(response.url(), response.profile(), FailCategory.FINAL_PROCESS_CONTEXT, e.getMessage(), -1);
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
return null;
}
final long parsingEndTime = System.currentTimeMillis();
- removed ZURL data structure; removed also the ZURL data file - replaced load failure logging by information which is stored in Solr - fixed a bug with crawling of feeds: added must-match pattern application to feed urls to filter out such urls which shall not be in a wanted domain - delegatedURLs, which also used ZURLs are now temporary objects in memory
2013-09-17 15:27:02 +02:00
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
// put anchors on crawl stack
final long stackStartTime = System.currentTimeMillis();
- removed ZURL data structure; removed also the ZURL data file - replaced load failure logging by information which is stored in Solr - fixed a bug with crawling of feeds: added must-match pattern application to feed urls to filter out such urls which shall not be in a wanted domain - delegatedURLs, which also used ZURLs are now temporary objects in memory
2013-09-17 15:27:02 +02:00
// check if the documents have valid urls; this is not a bug patch; it is possible that
// i.e. the result of a feed parsing results in documents from domains which shall be filtered by the crawl profile
if (response.profile() != null) {
ArrayList<Document> newDocs = new ArrayList<Document>();
for (Document doc: documents) {
added (partly commented) test code for url rewrite methods .. to be completed
2013-11-10 18:50:34 +01:00
//doc.rewrite_dc_source(rewritePattern, "");
fix for crawl domain counter limitation (limit was reached too early)
2013-09-26 13:41:52 +02:00
String rejectReason = this.crawlStacker.checkAcceptanceChangeable(doc.dc_source(), response.profile(), 1 /*depth is irrelevant here, we just make clear its not the start url*/);
- removed ZURL data structure; removed also the ZURL data file - replaced load failure logging by information which is stored in Solr - fixed a bug with crawling of feeds: added must-match pattern application to feed urls to filter out such urls which shall not be in a wanted domain - delegatedURLs, which also used ZURLs are now temporary objects in memory
2013-09-17 15:27:02 +02:00
if (rejectReason == null) {
newDocs.add(doc);
} else {
// we consider this as fail urls to have a tracking of the problem
if (rejectReason != null && !rejectReason.startsWith("double in")) {
self-healing of mistakenly deactivated crawl profiles. This fixes a bug which can happen in rare cases when a crawl start and a cleanup process happen at the same time.
2013-09-25 18:27:54 +02:00
final CrawlProfile profile = this.crawler.get(UTF8.getBytes(response.profile().handle()));
- removed ZURL data structure; removed also the ZURL data file - replaced load failure logging by information which is stored in Solr - fixed a bug with crawling of feeds: added must-match pattern application to feed urls to filter out such urls which shall not be in a wanted domain - delegatedURLs, which also used ZURLs are now temporary objects in memory
2013-09-17 15:27:02 +02:00
this.crawlStacker.nextQueue.errorURL.push(response.url(), profile, FailCategory.FINAL_LOAD_CONTEXT, rejectReason, -1);
}
}
}
if (newDocs.size() != documents.length) {
documents = (Document[]) newDocs.toArray();
}
}
// collect anchors within remaining documents
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
if ((processCase == EventOrigin.PROXY_LOAD || processCase == EventOrigin.LOCAL_CRAWLING) &&
(
response.profile() == null ||
response.depth() < response.profile().depth() ||
response.profile().crawlerNoDepthLimitMatchPattern().matcher(response.url().toNormalform(true)).matches()
)
) {
// get the hyperlinks
- the webgraph shall store all links which appear on a web page and not all unique links! This made it necessary, that a large portion of the parser and link processing classes must be adopted to carry a different type of link collection which carry a property attribute which are attached to web anchors. - introduction of a new URL class, AnchorURL - the other url classes, DigestURI and MultiProtocolURI had been renamed and refactored to fit into a new document package schema, document.id - cleanup of net.yacy.cora.document package and refactoring
2013-09-15 00:30:23 +02:00
final Map<DigestURL, String> hl = Document.getHyperlinks(documents);
load image only if their parser flag is activated
2013-11-04 11:59:28 +01:00
for (Map.Entry<DigestURL, String> entry: Document.getImagelinks(documents).entrySet()) {
if (TextParser.supportsExtension(entry.getKey()) == null) hl.put(entry.getKey(), entry.getValue());
}
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
// add all media links also to the crawl stack. They will be re-sorted to the NOLOAD queue and indexed afterwards as pure links
if (response.profile().directDocByURL()) {
load image only if their parser flag is activated
2013-11-04 11:59:28 +01:00
for (Map.Entry<DigestURL, String> entry: Document.getImagelinks(documents).entrySet()) {
if (TextParser.supportsExtension(entry.getKey()) != null) hl.put(entry.getKey(), entry.getValue());
}
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
hl.putAll(Document.getApplinks(documents));
hl.putAll(Document.getVideolinks(documents));
hl.putAll(Document.getAudiolinks(documents));
}
// insert those hyperlinks to the crawler
- the webgraph shall store all links which appear on a web page and not all unique links! This made it necessary, that a large portion of the parser and link processing classes must be adopted to carry a different type of link collection which carry a property attribute which are attached to web anchors. - introduction of a new URL class, AnchorURL - the other url classes, DigestURI and MultiProtocolURI had been renamed and refactored to fit into a new document package schema, document.id - cleanup of net.yacy.cora.document package and refactoring
2013-09-15 00:30:23 +02:00
MultiProtocolURL nextUrl;
for ( final Map.Entry<DigestURL, String> nextEntry : hl.entrySet() ) {
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
// check for interruption
checkInterruption();
// process the next hyperlink
nextUrl = nextEntry.getKey();
String u = nextUrl.toNormalform(true, true);
if ( !(u.startsWith("http://")
|| u.startsWith("https://")
|| u.startsWith("ftp://")
|| u.startsWith("smb://") || u.startsWith("file://")) ) {
continue;
}
// rewrite the url
String u0 = LibraryProvider.urlRewriter.apply(u);
if (!u.equals(u0)) {
- refactoring of log to ConcurrentLog: jdk-based logger tend to block at java.util.logging.Logger.log(Logger.java:476) in concurrent environments. This makes logging a main performance issue. To overcome this problem, this is a add-on to jdk logging to put log entries on a concurrent message queue and log the messages one by one using a separate process. - FTPClient uses the concurrent logging instead of the log4j logger
2013-07-09 14:28:25 +02:00
log.info("REWRITE of url = \"" + u + "\" to \"" + u0 + "\"");
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
u = u0;
}
added (partly commented) test code for url rewrite methods .. to be completed
2013-11-10 18:50:34 +01:00
//Matcher m = rewritePattern.matcher(u);
//if (m.matches()) u = m.replaceAll("");
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
// enqueue the hyperlink into the pre-notice-url db
try {
this.crawlStacker.enqueueEntry(new Request(
response.initiator(),
- the webgraph shall store all links which appear on a web page and not all unique links! This made it necessary, that a large portion of the parser and link processing classes must be adopted to carry a different type of link collection which carry a property attribute which are attached to web anchors. - introduction of a new URL class, AnchorURL - the other url classes, DigestURI and MultiProtocolURI had been renamed and refactored to fit into a new document package schema, document.id - cleanup of net.yacy.cora.document package and refactoring
2013-09-15 00:30:23 +02:00
new DigestURL(u),
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
response.url().hash(),
nextEntry.getValue(),
new Date(),
response.profile().handle(),
response.depth() + 1,
0,
0,
response.size() < 0 ? 0 : response.size()));
Added 'final' for all exception blocks as this helps the Java compiler to optimize memory usage Conflicts: source/net/yacy/search/Switchboard.java
2013-07-17 18:31:30 +02:00
} catch (final MalformedURLException e ) {
- refactoring of log to ConcurrentLog: jdk-based logger tend to block at java.util.logging.Logger.log(Logger.java:476) in concurrent environments. This makes logging a main performance issue. To overcome this problem, this is a add-on to jdk logging to put log entries on a concurrent message queue and log the messages one by one using a separate process. - FTPClient uses the concurrent logging instead of the log4j logger
2013-07-09 14:28:25 +02:00
ConcurrentLog.logException(e);
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
}
}
final long stackEndTime = System.currentTimeMillis();
if ( this.log.isInfo() ) {
- refactoring of log to ConcurrentLog: jdk-based logger tend to block at java.util.logging.Logger.log(Logger.java:476) in concurrent environments. This makes logging a main performance issue. To overcome this problem, this is a add-on to jdk logging to put log entries on a concurrent message queue and log the messages one by one using a separate process. - FTPClient uses the concurrent logging instead of the log4j logger
2013-07-09 14:28:25 +02:00
this.log.info("CRAWL: ADDED "
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
+ hl.size()
+ " LINKS FROM "
+ response.url().toNormalform(true)
+ ", STACKING TIME = "
+ (stackEndTime - stackStartTime)
+ ", PARSING TIME = "
+ (parsingEndTime - parsingStartTime));
}
}
return documents;
}
public IndexingQueueEntry condenseDocument(final IndexingQueueEntry in) {
in.queueEntry.updateStatus(Response.QUEUE_STATE_CONDENSING);
CrawlProfile profile = in.queueEntry.profile();
String urls = in.queueEntry.url().toNormalform(true);
// check profile attributes which prevent indexing (while crawling is allowed)
if (!profile.indexText() && !profile.indexMedia()) {
- refactoring of log to ConcurrentLog: jdk-based logger tend to block at java.util.logging.Logger.log(Logger.java:476) in concurrent environments. This makes logging a main performance issue. To overcome this problem, this is a add-on to jdk logging to put log entries on a concurrent message queue and log the messages one by one using a separate process. - FTPClient uses the concurrent logging instead of the log4j logger
2013-07-09 14:28:25 +02:00
if (this.log.isInfo()) this.log.info("Not Condensed Resource '" + urls + "': indexing of this media type not wanted by crawl profile");
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
return new IndexingQueueEntry(in.queueEntry, in.documents, null);
}
- added a new field for the regular expression in crawl start - added the field in crawl profile - adopted logging end error management - adopted duplicate document detection - added a new rule to the indexing process to reject non-matching content - full redesign of the expert crawl start servlet The new filter field can now be seen in /CrawlStartExpert_p.html at Section "Document Filter", subsection item "Filter on Content of Document"
2013-04-26 10:49:55 +02:00
if (!(profile.indexUrlMustMatchPattern() == CrawlProfile.MATCH_ALL_PATTERN || profile.indexUrlMustMatchPattern().matcher(urls).matches()) ||
(profile.indexUrlMustNotMatchPattern() != CrawlProfile.MATCH_NEVER_PATTERN && profile.indexUrlMustNotMatchPattern().matcher(urls).matches())) {
- refactoring of log to ConcurrentLog: jdk-based logger tend to block at java.util.logging.Logger.log(Logger.java:476) in concurrent environments. This makes logging a main performance issue. To overcome this problem, this is a add-on to jdk logging to put log entries on a concurrent message queue and log the messages one by one using a separate process. - FTPClient uses the concurrent logging instead of the log4j logger
2013-07-09 14:28:25 +02:00
if (this.log.isInfo()) this.log.info("Not Condensed Resource '" + urls + "': indexing prevented by regular expression on url; indexUrlMustMatchPattern = " + profile.indexUrlMustMatchPattern().pattern() + ", indexUrlMustNotMatchPattern = " + profile.indexUrlMustNotMatchPattern().pattern());
- removed ZURL data structure; removed also the ZURL data file - replaced load failure logging by information which is stored in Solr - fixed a bug with crawling of feeds: added must-match pattern application to feed urls to filter out such urls which shall not be in a wanted domain - delegatedURLs, which also used ZURLs are now temporary objects in memory
2013-09-17 15:27:02 +02:00
// create a new errorURL DB entry
this.crawlQueues.errorURL.push(in.queueEntry.url(), profile, FailCategory.FINAL_PROCESS_CONTEXT, "indexing prevented by regular expression on url; indexUrlMustMatchPattern = " + profile.indexUrlMustMatchPattern().pattern() + ", indexUrlMustNotMatchPattern = " + profile.indexUrlMustNotMatchPattern().pattern(), -1);
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
return new IndexingQueueEntry(in.queueEntry, in.documents, null);
}
// check which files may take part in the indexing process
final List<Document> doclist = new ArrayList<Document>();
- added a new field for the regular expression in crawl start - added the field in crawl profile - adopted logging end error management - adopted duplicate document detection - added a new rule to the indexing process to reject non-matching content - full redesign of the expert crawl start servlet The new filter field can now be seen in /CrawlStartExpert_p.html at Section "Document Filter", subsection item "Filter on Content of Document"
2013-04-26 10:49:55 +02:00
docloop: for (final Document document : in.documents) {
added a crawl option to obey html-meta-robots-noindex. This is on by default.
2013-07-03 14:50:06 +02:00
if (document.indexingDenied() && profile.obeyHtmlRobotsNoindex()) {
- refactoring of log to ConcurrentLog: jdk-based logger tend to block at java.util.logging.Logger.log(Logger.java:476) in concurrent environments. This makes logging a main performance issue. To overcome this problem, this is a add-on to jdk logging to put log entries on a concurrent message queue and log the messages one by one using a separate process. - FTPClient uses the concurrent logging instead of the log4j logger
2013-07-09 14:28:25 +02:00
if (this.log.isInfo()) this.log.info("Not Condensed Resource '" + urls + "': denied by document-attached noindexing rule");
- removed ZURL data structure; removed also the ZURL data file - replaced load failure logging by information which is stored in Solr - fixed a bug with crawling of feeds: added must-match pattern application to feed urls to filter out such urls which shall not be in a wanted domain - delegatedURLs, which also used ZURLs are now temporary objects in memory
2013-09-17 15:27:02 +02:00
// create a new errorURL DB entry
this.crawlQueues.errorURL.push(in.queueEntry.url(), profile, FailCategory.FINAL_PROCESS_CONTEXT, "denied by document-attached noindexing rule", -1);
- added a new field for the regular expression in crawl start - added the field in crawl profile - adopted logging end error management - adopted duplicate document detection - added a new rule to the indexing process to reject non-matching content - full redesign of the expert crawl start servlet The new filter field can now be seen in /CrawlStartExpert_p.html at Section "Document Filter", subsection item "Filter on Content of Document"
2013-04-26 10:49:55 +02:00
continue docloop;
}
if (!(profile.indexContentMustMatchPattern() == CrawlProfile.MATCH_ALL_PATTERN || profile.indexContentMustMatchPattern().matcher(document.getTextString()).matches()) ||
(profile.indexContentMustNotMatchPattern() != CrawlProfile.MATCH_NEVER_PATTERN && profile.indexContentMustNotMatchPattern().matcher(document.getTextString()).matches())) {
- refactoring of log to ConcurrentLog: jdk-based logger tend to block at java.util.logging.Logger.log(Logger.java:476) in concurrent environments. This makes logging a main performance issue. To overcome this problem, this is a add-on to jdk logging to put log entries on a concurrent message queue and log the messages one by one using a separate process. - FTPClient uses the concurrent logging instead of the log4j logger
2013-07-09 14:28:25 +02:00
if (this.log.isInfo()) this.log.info("Not Condensed Resource '" + urls + "': indexing prevented by regular expression on content; indexContentMustMatchPattern = " + profile.indexContentMustMatchPattern().pattern() + ", indexContentMustNotMatchPattern = " + profile.indexContentMustNotMatchPattern().pattern());
- removed ZURL data structure; removed also the ZURL data file - replaced load failure logging by information which is stored in Solr - fixed a bug with crawling of feeds: added must-match pattern application to feed urls to filter out such urls which shall not be in a wanted domain - delegatedURLs, which also used ZURLs are now temporary objects in memory
2013-09-17 15:27:02 +02:00
// create a new errorURL DB entry
this.crawlQueues.errorURL.push(in.queueEntry.url(), profile, FailCategory.FINAL_PROCESS_CONTEXT, "indexing prevented by regular expression on content; indexContentMustMatchPattern = " + profile.indexContentMustMatchPattern().pattern() + ", indexContentMustNotMatchPattern = " + profile.indexContentMustNotMatchPattern().pattern(), -1);
- added a new field for the regular expression in crawl start - added the field in crawl profile - adopted logging end error management - adopted duplicate document detection - added a new rule to the indexing process to reject non-matching content - full redesign of the expert crawl start servlet The new filter field can now be seen in /CrawlStartExpert_p.html at Section "Document Filter", subsection item "Filter on Content of Document"
2013-04-26 10:49:55 +02:00
continue docloop;
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
}
doclist.add(document);
}
if ( doclist.isEmpty() ) {
return new IndexingQueueEntry(in.queueEntry, in.documents, null);
}
in.documents = doclist.toArray(new Document[doclist.size()]);
final Condenser[] condenser = new Condenser[in.documents.length];
for ( int i = 0; i < in.documents.length; i++ ) {
condenser[i] =
new Condenser(
in.documents[i], in.queueEntry.profile().indexText(),
in.queueEntry.profile().indexMedia(),
LibraryProvider.dymLib, LibraryProvider.synonyms, true);
// update image result list statistics
// its good to do this concurrently here, because it needs a DNS lookup
// to compute a URL hash which is necessary for a double-check
ResultImages.registerImages(in.queueEntry.url(), in.documents[i], (profile == null)
? true
: !profile.remoteIndexing());
}
return new IndexingQueueEntry(in.queueEntry, in.documents, condenser);
}
public IndexingQueueEntry webStructureAnalysis(final IndexingQueueEntry in) {
in.queueEntry.updateStatus(Response.QUEUE_STATE_STRUCTUREANALYSIS);
for (Document document : in.documents) {
assert this.webStructure != null;
assert in != null;
assert in.queueEntry != null;
assert in.documents != null;
assert in.queueEntry != null;
this.webStructure.generateCitationReference(in.queueEntry.url(), document); // [outlinksSame, outlinksOther]
}
return in;
}
public void storeDocumentIndex(final IndexingQueueEntry in) {
in.queueEntry.updateStatus(Response.QUEUE_STATE_INDEXSTORAGE);
// the condenser may be null in case that an indexing is not wanted (there may be a no-indexing flag in the file)
if ( in.condenser != null ) {
for ( int i = 0; i < in.documents.length; i++ ) {
added the new field harvestkey_s to the collection index and the webgraph index which is temporary filled with the crawl profile key. This is used to select a set of documents for post-processing as soon as a crawl is finished. Now the postprocessing for a specific crawl is started when that specific crawl is finished and not at the end of all post-processing steps.
2013-09-25 14:38:24 +02:00
CrawlProfile profile = in.queueEntry.profile();
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
storeDocumentIndex(
in.queueEntry,
added collection attribute also to the rss feed reader
2013-04-24 01:14:35 +02:00
in.queueEntry.profile().collections(),
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
in.documents[i],
in.condenser[i],
null,
added the new field harvestkey_s to the collection index and the webgraph index which is temporary filled with the crawl profile key. This is used to select a set of documents for post-processing as soon as a crawl is finished. Now the postprocessing for a specific crawl is started when that specific crawl is finished and not at the end of all post-processing steps.
2013-09-25 14:38:24 +02:00
profile == null ? "crawler" : profile.handle());
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
}
}
in.queueEntry.updateStatus(Response.QUEUE_STATE_FINISHED);
}
added the new field harvestkey_s to the collection index and the webgraph index which is temporary filled with the crawl profile key. This is used to select a set of documents for post-processing as soon as a crawl is finished. Now the postprocessing for a specific crawl is started when that specific crawl is finished and not at the end of all post-processing steps.
2013-09-25 14:38:24 +02:00
/**
*
* @param queueEntry
* @param collections
* @param document
* @param condenser
* @param searchEvent
* @param sourceName if this document was created by a crawl, then the sourceName contains the crawl hash
*/
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
private void storeDocumentIndex(
final Response queueEntry,
added a feature to set a collection for a crawl result based on a regular expression on th url: the collection attribut for a crawl start may be now either a token or a list of tokens, seperated by ',' where a token is either a string or a pair <string,pattern> where the string is separated to the pattern with a ':' and the string is assigned to the document as collection only if the pattern matches with the url.
2013-08-25 00:13:48 +02:00
final Map<String, Pattern> collections,
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
final Document document,
final Condenser condenser,
final SearchEvent searchEvent,
final String sourceName) {
//TODO: document must carry referer, size and last modified
// CREATE INDEX
final String dc_title = document.dc_title();
- the webgraph shall store all links which appear on a web page and not all unique links! This made it necessary, that a large portion of the parser and link processing classes must be adopted to carry a different type of link collection which carry a property attribute which are attached to web anchors. - introduction of a new URL class, AnchorURL - the other url classes, DigestURI and MultiProtocolURI had been renamed and refactored to fit into a new document package schema, document.id - cleanup of net.yacy.cora.document package and refactoring
2013-09-15 00:30:23 +02:00
final DigestURL url = document.dc_source();
final DigestURL referrerURL = queueEntry.referrerURL();
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
EventOrigin processCase = queueEntry.processCase(this.peers.mySeed().hash);
added a crawl option to obey html-meta-robots-noindex. This is on by default.
2013-07-03 14:50:06 +02:00
CrawlProfile profile = queueEntry.profile();
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
added a crawl option to obey html-meta-robots-noindex. This is on by default.
2013-07-03 14:50:06 +02:00
if (condenser == null || (document.indexingDenied() && profile.obeyHtmlRobotsNoindex())) {
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
//if (this.log.isInfo()) log.logInfo("Not Indexed Resource '" + queueEntry.url().toNormalform(false, true) + "': denied by rule in document, process case=" + processCase);
- removed ZURL data structure; removed also the ZURL data file - replaced load failure logging by information which is stored in Solr - fixed a bug with crawling of feeds: added must-match pattern application to feed urls to filter out such urls which shall not be in a wanted domain - delegatedURLs, which also used ZURLs are now temporary objects in memory
2013-09-17 15:27:02 +02:00
// create a new errorURL DB entry
this.crawlQueues.errorURL.push(url, profile, FailCategory.FINAL_PROCESS_CONTEXT, "denied by rule in document, process case=" + processCase, -1);
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
return;
}
Fix for NPE on shutdown: java.lang.NullPointerException at net.yacy.search.Switchboard.storeDocumentIndex(Switchboard.java:2732) at net.yacy.search.Switchboard.access00(Switchboard.java:207) at net.yacy.search.Switchboard.run(Switchboard.java:3049)
2013-07-25 12:02:11 +02:00
if ( profile != null && !profile.indexText() && !profile.indexMedia() ) {
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
//if (this.log.isInfo()) log.logInfo("Not Indexed Resource '" + queueEntry.url().toNormalform(false, true) + "': denied by profile rule, process case=" + processCase + ", profile name = " + queueEntry.profile().name());
- removed ZURL data structure; removed also the ZURL data file - replaced load failure logging by information which is stored in Solr - fixed a bug with crawling of feeds: added must-match pattern application to feed urls to filter out such urls which shall not be in a wanted domain - delegatedURLs, which also used ZURLs are now temporary objects in memory
2013-09-17 15:27:02 +02:00
// create a new errorURL DB entry
this.crawlQueues.errorURL.push(url, profile, FailCategory.FINAL_LOAD_CONTEXT, "denied by profile rule, process case="
+ processCase
+ ", profile name = "
+ profile.collectionName(), -1);
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
return;
}
// remove stopwords
- refactoring of log to ConcurrentLog: jdk-based logger tend to block at java.util.logging.Logger.log(Logger.java:476) in concurrent environments. This makes logging a main performance issue. To overcome this problem, this is a add-on to jdk logging to put log entries on a concurrent message queue and log the messages one by one using a separate process. - FTPClient uses the concurrent logging instead of the log4j logger
2013-07-09 14:28:25 +02:00
this.log.info("Excluded " + condenser.excludeWords(stopwords) + " words in URL " + url);
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
// STORE WORD INDEX
SolrInputDocument newEntry =
this.index.storeDocument(
url,
referrerURL,
added collection attribute also to the rss feed reader
2013-04-24 01:14:35 +02:00
collections,
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
queueEntry.getResponseHeader(),
document,
condenser,
searchEvent,
sourceName,
fix for http://forum.yacy-websuche.de/viewtopic.php?f=5&t=4652 generate dht data even if dht receive and dht transmission is switched off
2013-05-08 16:48:45 +02:00
getConfigBool(SwitchboardConstants.DHT_ENABLED, false));
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
final RSSFeed feed =
EventChannel.channels(queueEntry.initiator() == null
? EventChannel.PROXY
: Base64Order.enhancedCoder.equal(
queueEntry.initiator(),
ASCII.getBytes(this.peers.mySeed().hash))
? EventChannel.LOCALINDEXING
: EventChannel.REMOTEINDEXING);
feed.addMessage(new RSSMessage("Indexed web page", dc_title, queueEntry.url(), ASCII.String(queueEntry.url().hash())));
// store rss feeds in document into rss table
- the webgraph shall store all links which appear on a web page and not all unique links! This made it necessary, that a large portion of the parser and link processing classes must be adopted to carry a different type of link collection which carry a property attribute which are attached to web anchors. - introduction of a new URL class, AnchorURL - the other url classes, DigestURI and MultiProtocolURI had been renamed and refactored to fit into a new document package schema, document.id - cleanup of net.yacy.cora.document package and refactoring
2013-09-15 00:30:23 +02:00
for ( final Map.Entry<DigestURL, String> rssEntry : document.getRSS().entrySet() ) {
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
final Tables.Data rssRow = new Tables.Data();
rssRow.put("referrer", url.hash());
rssRow.put("url", UTF8.getBytes(rssEntry.getKey().toNormalform(true)));
rssRow.put("title", UTF8.getBytes(rssEntry.getValue()));
rssRow.put("recording_date", new Date());
try {
added the generation of 50 (!!) new solr field in the core 'webgraph'. The default schema uses only some of them and the resting search index has now the following properties: - webgraph size will have about 40 times as much entries as default index - the complete index size will increase and may be about the double size of current amount As testing showed, not much indexing performance is lost. The default index will be smaller (moved fields out of it); thus searching can be faster. The new index will cause that some old parts in YaCy can be removed, i.e. specialized webgraph data and the noload crawler. The new index will make it possible to: - search within link texts of linked but not indexed documents (about 20 times of document index in size!!) - get a very detailed link graph - enhance ranking using a complete link graph To get the full access to the new index, the API to solr has now two access points: one with attribute core=collection1 for the default search index and core=webgraph to the new webgraph search index. This is also avaiable for p2p operation but client access is not yet implemented.
2013-02-22 15:45:15 +01:00
this.tables.update("rss", rssEntry.getKey().hash(), rssRow);
Added 'final' for all exception blocks as this helps the Java compiler to optimize memory usage Conflicts: source/net/yacy/search/Switchboard.java
2013-07-17 18:31:30 +02:00
} catch (final IOException e ) {
- refactoring of log to ConcurrentLog: jdk-based logger tend to block at java.util.logging.Logger.log(Logger.java:476) in concurrent environments. This makes logging a main performance issue. To overcome this problem, this is a add-on to jdk logging to put log entries on a concurrent message queue and log the messages one by one using a separate process. - FTPClient uses the concurrent logging instead of the log4j logger
2013-07-09 14:28:25 +02:00
ConcurrentLog.logException(e);
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
}
}
// update url result list statistics
ResultURLs.stack(
ASCII.String(url.hash()), // loaded url db entry
url.getHost(),
queueEntry.initiator(), // initiator peer hash
UTF8.getBytes(this.peers.mySeed().hash), // executor peer hash
processCase // process case
);
// increment number of indexed urls
this.indexedPages++;
// update profiling info
if ( System.currentTimeMillis() - lastPPMUpdate > 20000 ) {
// we don't want to do this too often
updateMySeed();
EventTracker.update(EventTracker.EClass.PPM, Long.valueOf(currentPPM()), true);
lastPPMUpdate = System.currentTimeMillis();
}
EventTracker.update(EventTracker.EClass.INDEX, url.toNormalform(true), false);
// if this was performed for a remote crawl request, notify requester
if ( (processCase == EventOrigin.GLOBAL_CRAWLING) && (queueEntry.initiator() != null) ) {
final Seed initiatorPeer = this.peers.get(ASCII.String(queueEntry.initiator()));
if ( initiatorPeer != null ) {
if ( this.clusterhashes != null ) {
initiatorPeer.setAlternativeAddress(this.clusterhashes.get(queueEntry.initiator()));
}
// start a thread for receipt sending to avoid a blocking here
made index storage from DHT search result concurrently. This prevents blocking by high CPU usage during search. Also: removed query from Solr for DHT search results; results are taken from the pending queue.
2013-03-02 10:25:52 +01:00
SolrDocument sd = this.index.fulltext().getDefaultConfiguration().toSolrDocument(newEntry);
new Thread(new receiptSending(initiatorPeer, new URIMetadataNode(sd)), "sending receipt to " + ASCII.String(queueEntry.initiator())).start();
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
}
}
}
public final void addAllToIndex(
- the webgraph shall store all links which appear on a web page and not all unique links! This made it necessary, that a large portion of the parser and link processing classes must be adopted to carry a different type of link collection which carry a property attribute which are attached to web anchors. - introduction of a new URL class, AnchorURL - the other url classes, DigestURI and MultiProtocolURI had been renamed and refactored to fit into a new document package schema, document.id - cleanup of net.yacy.cora.document package and refactoring
2013-09-15 00:30:23 +02:00
final DigestURL url,
final Map<DigestURL, String> links,
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
final SearchEvent searchEvent,
added collection attribute also to the rss feed reader
2013-04-24 01:14:35 +02:00
final String heuristicName,
added a feature to set a collection for a crawl result based on a regular expression on th url: the collection attribut for a crawl start may be now either a token or a list of tokens, seperated by ',' where a token is either a string or a pair <string,pattern> where the string is separated to the pattern with a ':' and the string is assigned to the document as collection only if the pattern matches with the url.
2013-08-25 00:13:48 +02:00
final Map<String, Pattern> collections) {
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
- the webgraph shall store all links which appear on a web page and not all unique links! This made it necessary, that a large portion of the parser and link processing classes must be adopted to carry a different type of link collection which carry a property attribute which are attached to web anchors. - introduction of a new URL class, AnchorURL - the other url classes, DigestURI and MultiProtocolURI had been renamed and refactored to fit into a new document package schema, document.id - cleanup of net.yacy.cora.document package and refactoring
2013-09-15 00:30:23 +02:00
List<DigestURL> urls = new ArrayList<DigestURL>();
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
// add the landing page to the index. should not load that again since it should be in the cache
redesign of index.exist-test: this shall now not be done using a single id to be tested, but with a collection of ids. This will cause only a single call to solr instead of many. The result is a much better performace when testing the existence of many urls. The effect should cause very much less IO during index transmission, both on sender and receiver side.
2013-05-17 13:59:37 +02:00
if (url != null) {
urls.add(url);
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
}
// check if some of the links match with the query
- the webgraph shall store all links which appear on a web page and not all unique links! This made it necessary, that a large portion of the parser and link processing classes must be adopted to carry a different type of link collection which carry a property attribute which are attached to web anchors. - introduction of a new URL class, AnchorURL - the other url classes, DigestURI and MultiProtocolURI had been renamed and refactored to fit into a new document package schema, document.id - cleanup of net.yacy.cora.document package and refactoring
2013-09-15 00:30:23 +02:00
final Map<DigestURL, String> matcher = searchEvent.query.separateMatches(links);
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
// take the matcher and load them all
- the webgraph shall store all links which appear on a web page and not all unique links! This made it necessary, that a large portion of the parser and link processing classes must be adopted to carry a different type of link collection which carry a property attribute which are attached to web anchors. - introduction of a new URL class, AnchorURL - the other url classes, DigestURI and MultiProtocolURI had been renamed and refactored to fit into a new document package schema, document.id - cleanup of net.yacy.cora.document package and refactoring
2013-09-15 00:30:23 +02:00
for (final Map.Entry<DigestURL, String> entry : matcher.entrySet()) {
urls.add(new DigestURL(entry.getKey(), (byte[]) null));
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
}
// take then the no-matcher and load them also
- the webgraph shall store all links which appear on a web page and not all unique links! This made it necessary, that a large portion of the parser and link processing classes must be adopted to carry a different type of link collection which carry a property attribute which are attached to web anchors. - introduction of a new URL class, AnchorURL - the other url classes, DigestURI and MultiProtocolURI had been renamed and refactored to fit into a new document package schema, document.id - cleanup of net.yacy.cora.document package and refactoring
2013-09-15 00:30:23 +02:00
for (final Map.Entry<DigestURL, String> entry : links.entrySet()) {
urls.add(new DigestURL(entry.getKey(), (byte[]) null));
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
}
redesign of index.exist-test: this shall now not be done using a single id to be tested, but with a collection of ids. This will cause only a single call to solr instead of many. The result is a much better performace when testing the existence of many urls. The effect should cause very much less IO during index transmission, both on sender and receiver side.
2013-05-17 13:59:37 +02:00
addToIndex(urls, searchEvent, heuristicName, collections);
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
}
fixes to deletion methods (removed unnecessary concurrency and added removal of crawl queue entries)
2013-05-08 13:26:25 +02:00
public void remove(final Collection<String> deleteIDs) {
this.index.fulltext().remove(deleteIDs);
for (String id: deleteIDs) {
refactoring (im preparation for new removeHost method)
2013-09-05 09:59:41 +02:00
this.crawlQueues.removeURL(ASCII.getBytes(id));
fixes to deletion methods (removed unnecessary concurrency and added removal of crawl queue entries)
2013-05-08 13:26:25 +02:00
}
}
public void remove(final byte[] urlhash) {
this.index.fulltext().remove(urlhash);
refactoring (im preparation for new removeHost method)
2013-09-05 09:59:41 +02:00
this.crawlQueues.removeURL(urlhash);
fixes to deletion methods (removed unnecessary concurrency and added removal of crawl queue entries)
2013-05-08 13:26:25 +02:00
}
- the webgraph shall store all links which appear on a web page and not all unique links! This made it necessary, that a large portion of the parser and link processing classes must be adopted to carry a different type of link collection which carry a property attribute which are attached to web anchors. - introduction of a new URL class, AnchorURL - the other url classes, DigestURI and MultiProtocolURI had been renamed and refactored to fit into a new document package schema, document.id - cleanup of net.yacy.cora.document package and refactoring
2013-09-15 00:30:23 +02:00
public void stackURLs(Set<DigestURL> rootURLs, final CrawlProfile profile, final Set<DigestURL> successurls, final Map<DigestURL,String> failurls) {
prevent that crawl starts with very large url lists cause a time-out in the user front-end
2013-02-15 01:58:28 +01:00
if (rootURLs == null || rootURLs.size() == 0) return;
some dns-timing changes: since httpclient uses the domain-cache it is useful not to clean the domain cache until crawling is running (domains are filled into this cache) On huge crawl-starts (eg. from file) my DNS did not follow the high rates - so I reduced the rate and give some more time(-out)
2013-09-25 15:01:28 +02:00
final List<Thread> stackthreads = new ArrayList<Thread>(); // do this concurrently
- the webgraph shall store all links which appear on a web page and not all unique links! This made it necessary, that a large portion of the parser and link processing classes must be adopted to carry a different type of link collection which carry a property attribute which are attached to web anchors. - introduction of a new URL class, AnchorURL - the other url classes, DigestURI and MultiProtocolURI had been renamed and refactored to fit into a new document package schema, document.id - cleanup of net.yacy.cora.document package and refactoring
2013-09-15 00:30:23 +02:00
for (DigestURL url: rootURLs) {
final DigestURL turl = url;
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
Thread t = new Thread() {
public void run() {
String failreason;
if ((failreason = Switchboard.this.stackUrl(profile, turl)) == null) successurls.add(turl); else failurls.put(turl, failreason);
}
};
t.start();
stackthreads.add(t);
some dns-timing changes: since httpclient uses the domain-cache it is useful not to clean the domain cache until crawling is running (domains are filled into this cache) On huge crawl-starts (eg. from file) my DNS did not follow the high rates - so I reduced the rate and give some more time(-out)
2013-09-25 15:01:28 +02:00
try {Thread.sleep(100);} catch (final InterruptedException e) {} // to prevent that this fires more than 10 connections pre second!
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
}
some dns-timing changes: since httpclient uses the domain-cache it is useful not to clean the domain cache until crawling is running (domains are filled into this cache) On huge crawl-starts (eg. from file) my DNS did not follow the high rates - so I reduced the rate and give some more time(-out)
2013-09-25 15:01:28 +02:00
final long waitingtime = 10 + (30000 / rootURLs.size()); // at most wait only halve an minute to prevent that the crawl start runs into a time-out
Added 'final' for all exception blocks as this helps the Java compiler to optimize memory usage Conflicts: source/net/yacy/search/Switchboard.java
2013-07-17 18:31:30 +02:00
for (Thread t: stackthreads) try {t.join(waitingtime);} catch (final InterruptedException e) {}
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
}
/**
* stack the url to the crawler
* @param profile
* @param url
* @return null if this was ok. If this failed, return a string with a fail reason
*/
- the webgraph shall store all links which appear on a web page and not all unique links! This made it necessary, that a large portion of the parser and link processing classes must be adopted to carry a different type of link collection which carry a property attribute which are attached to web anchors. - introduction of a new URL class, AnchorURL - the other url classes, DigestURI and MultiProtocolURI had been renamed and refactored to fit into a new document package schema, document.id - cleanup of net.yacy.cora.document package and refactoring
2013-09-15 00:30:23 +02:00
public String stackUrl(CrawlProfile profile, DigestURL url) {
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
byte[] handle = ASCII.getBytes(profile.handle());
// remove url from the index to be prepared for a re-crawl
final byte[] urlhash = url.hash();
fixes to deletion methods (removed unnecessary concurrency and added removal of crawl queue entries)
2013-05-08 13:26:25 +02:00
remove(urlhash);
prevent that concurrent deletion process causes wrong double-check in crawl start
2013-05-12 21:37:45 +02:00
// because the removal is done concurrenlty, it is possible that the crawl
// stacking may fail because of double occurrences of that url. Therefore
// we must wait here until the url has actually disappeared
int t = 100;
less overhead calling exist() with only one hash
2013-11-04 09:37:31 +01:00
Set<String> ids = new HashSet<String>(1); ids.add(ASCII.String(urlhash));
fix in forced deletion: forced commit needed
2013-07-25 09:53:19 +02:00
while (t-- > 0 && this.index.exists(ids).size() > 0) {
Added 'final' for all exception blocks as this helps the Java compiler to optimize memory usage Conflicts: source/net/yacy/search/Switchboard.java
2013-07-17 18:31:30 +02:00
try {Thread.sleep(100);} catch (final InterruptedException e) {}
- refactoring of log to ConcurrentLog: jdk-based logger tend to block at java.util.logging.Logger.log(Logger.java:476) in concurrent environments. This makes logging a main performance issue. To overcome this problem, this is a add-on to jdk logging to put log entries on a concurrent message queue and log the messages one by one using a separate process. - FTPClient uses the concurrent logging instead of the log4j logger
2013-07-09 14:28:25 +02:00
ConcurrentLog.fine("Switchboard", "STACKURL: waiting for deletion, t=" + t);
added tools to visualize image generation (i.e. during testing)
2013-09-09 12:58:26 +02:00
//if (t == 20) this.index.fulltext().commit(true);
if (t == 1) this.index.fulltext().commit(false);
prevent that concurrent deletion process causes wrong double-check in crawl start
2013-05-12 21:37:45 +02:00
}
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
// special handling of ftp protocol
if (url.isFTP()) {
try {
this.crawler.putActive(handle, profile);
ftp client shall be able to open non-anonymous ftp servers if login details are given
2013-12-28 22:42:02 +01:00
String userInfo = url.getUserInfo();
int p = userInfo == null ? -1 : userInfo.indexOf(':');
String user = userInfo == null ? FTPClient.ANONYMOUS : userInfo.substring(0, p);
String pw = userInfo == null || p == -1 ? "anomic" : userInfo.substring(p + 1);
this.crawlStacker.enqueueEntriesFTP(this.peers.mySeed().hash.getBytes(), profile.handle(), url.getHost(), url.getPort(), user, pw, false);
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
return null;
} catch (final Exception e) {
// mist
- refactoring of log to ConcurrentLog: jdk-based logger tend to block at java.util.logging.Logger.log(Logger.java:476) in concurrent environments. This makes logging a main performance issue. To overcome this problem, this is a add-on to jdk logging to put log entries on a concurrent message queue and log the messages one by one using a separate process. - FTPClient uses the concurrent logging instead of the log4j logger
2013-07-09 14:28:25 +02:00
ConcurrentLog.logException(e);
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
return "problem crawling an ftp site: " + e.getMessage();
}
}
fix for a problem which ocurres when a site is crawled where the start url is redirected.
2013-09-04 16:00:47 +02:00
// remove the document from the error-db
forced deletion of ZURL entries for a specific host for each host that appears in the crawl url list
2013-09-05 13:22:16 +02:00
byte[] hosthash = new byte[6]; System.arraycopy(urlhash, 6, hosthash, 0, 6);
Enhanced crawl start for very, very large crawl lists (i.e. > 5000) which had a problem because of badly used concurrency. This fix also caused a redesign of the whole host deletion process. This should fix bug http://bugs.yacy.net/view.php?id=250
2013-10-24 16:20:20 +02:00
Set<String> hosthashes = new HashSet<String>();
hosthashes.add(ASCII.String(hosthash));
this.crawlQueues.errorURL.removeHosts(hosthashes);
better and consistent deletions for error urls
2013-09-17 15:52:57 +02:00
this.index.fulltext().remove(urlhash);
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
// get a scraper to get the title
Document scraper;
try {
Redesign of crawler identification and robots steering. A non-p2p user in intranets and the internet can now choose to appear as Googlebot. This is an essential necessity to be able to compete in the field of commercial search appliances, since most web pages are these days optimized only for Google and no other search platform any more. All commercial search engine providers have a built-in fake-Google User Agent to be able to get the same search index as Google can do. Without the resistance against obeying to robots.txt in this case, no competition is possible any more. YaCy will always obey the robots.txt when it is used for crawling the web in a peer-to-peer network, but to establish a Search Appliance (like a Google Search Appliance, GSA) it is necessary to be able to behave exactly like a Google crawler. With this change, you will be able to switch the user agent when portal or intranet mode is selected on per-crawl-start basis. Every crawl start can have a different user agent.
2013-08-22 14:23:47 +02:00
scraper = this.loader.loadDocument(url, CacheStrategy.IFFRESH, BlacklistType.CRAWLER, profile.getAgent());
Added 'final' for all exception blocks as this helps the Java compiler to optimize memory usage Conflicts: source/net/yacy/search/Switchboard.java
2013-07-17 18:31:30 +02:00
} catch (final IOException e) {
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
return "scraper cannot load URL: " + e.getMessage();
}
final String title = scraper == null ? url.toNormalform(true) : scraper.dc_title();
replaced the single-text description solr field with a multi-value description_txt text field
2013-07-30 12:48:57 +02:00
final String description = scraper.dc_description().length > 0 ? scraper.dc_description()[0] : "";
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
// add the url to the crawl stack
this.crawler.removePassive(handle); // if there is an old entry, delete it
this.crawler.putActive(handle, profile);
final String reasonString = this.crawlStacker.stackCrawl(new Request(
this.peers.mySeed().hash.getBytes(),
url,
null,
"CRAWLING-ROOT",
new Date(),
profile.handle(),
0,
0,
0,
0
));
if (reasonString != null) return reasonString;
// create a bookmark from crawl start url
//final Set<String> tags=ListManager.string2set(BookmarkHelper.cleanTagsString(post.get("bookmarkFolder","/crawlStart")));
final Set<String> tags=ListManager.string2set(BookmarkHelper.cleanTagsString("/crawlStart"));
tags.add("crawlStart");
final String[] keywords = scraper.dc_subject();
if (keywords != null) {
for (final String k: keywords) {
final String kk = BookmarkHelper.cleanTagsString(k);
if (kk.length() > 0) tags.add(kk);
}
}
String tagStr = tags.toString();
if (tagStr.length() > 2 && tagStr.startsWith("[") && tagStr.endsWith("]")) tagStr = tagStr.substring(1, tagStr.length() - 2);
// we will create always a bookmark to use this to track crawled hosts
final BookmarksDB.Bookmark bookmark = this.bookmarksDB.createBookmark(url.toNormalform(true), "admin");
if (bookmark != null) {
bookmark.setProperty(BookmarksDB.Bookmark.BOOKMARK_TITLE, title);
bookmark.setProperty(BookmarksDB.Bookmark.BOOKMARK_DESCRIPTION, description);
bookmark.setOwner("admin");
bookmark.setPublic(false);
bookmark.setTags(tags, true);
this.bookmarksDB.saveBookmark(bookmark);
}
// do the same for ymarks
// TODO: could a non admin user add crawls?
try {
Redesign of crawler identification and robots steering. A non-p2p user in intranets and the internet can now choose to appear as Googlebot. This is an essential necessity to be able to compete in the field of commercial search appliances, since most web pages are these days optimized only for Google and no other search platform any more. All commercial search engine providers have a built-in fake-Google User Agent to be able to get the same search index as Google can do. Without the resistance against obeying to robots.txt in this case, no competition is possible any more. YaCy will always obey the robots.txt when it is used for crawling the web in a peer-to-peer network, but to establish a Search Appliance (like a Google Search Appliance, GSA) it is necessary to be able to behave exactly like a Google crawler. With this change, you will be able to switch the user agent when portal or intranet mode is selected on per-crawl-start basis. Every crawl start can have a different user agent.
2013-08-22 14:23:47 +02:00
this.tables.bookmarks.createBookmark(this.loader, url, profile.getAgent(), YMarkTables.USER_ADMIN, true, "crawlStart", "/Crawl Start");
Added 'final' for all exception blocks as this helps the Java compiler to optimize memory usage Conflicts: source/net/yacy/search/Switchboard.java
2013-07-17 18:31:30 +02:00
} catch (final IOException e) {
- refactoring of log to ConcurrentLog: jdk-based logger tend to block at java.util.logging.Logger.log(Logger.java:476) in concurrent environments. This makes logging a main performance issue. To overcome this problem, this is a add-on to jdk logging to put log entries on a concurrent message queue and log the messages one by one using a separate process. - FTPClient uses the concurrent logging instead of the log4j logger
2013-07-09 14:28:25 +02:00
ConcurrentLog.logException(e);
Added 'final' for all exception blocks as this helps the Java compiler to optimize memory usage Conflicts: source/net/yacy/search/Switchboard.java
2013-07-17 18:31:30 +02:00
} catch (final Failure e) {
- refactoring of log to ConcurrentLog: jdk-based logger tend to block at java.util.logging.Logger.log(Logger.java:476) in concurrent environments. This makes logging a main performance issue. To overcome this problem, this is a add-on to jdk logging to put log entries on a concurrent message queue and log the messages one by one using a separate process. - FTPClient uses the concurrent logging instead of the log4j logger
2013-07-09 14:28:25 +02:00
ConcurrentLog.logException(e);
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
}
// that was ok
return null;
}
/**
* load the content of a URL, parse the content and add the content to the index This process is started
* concurrently. The method returns immediately after the call.
*
* @param url the url that shall be indexed
* @param searchEvent (optional) a search event that shall get results from the indexed pages directly
* feeded. If object is null then it is ignored
* @throws IOException
* @throws Parser.Failure
*/
- the webgraph shall store all links which appear on a web page and not all unique links! This made it necessary, that a large portion of the parser and link processing classes must be adopted to carry a different type of link collection which carry a property attribute which are attached to web anchors. - introduction of a new URL class, AnchorURL - the other url classes, DigestURI and MultiProtocolURI had been renamed and refactored to fit into a new document package schema, document.id - cleanup of net.yacy.cora.document package and refactoring
2013-09-15 00:30:23 +02:00
public void addToIndex(final Collection<DigestURL> urls, final SearchEvent searchEvent, final String heuristicName, final Map<String, Pattern> collections) {
Map<String, DigestURL> urlmap = new HashMap<String, DigestURL>();
for (DigestURL url: urls) urlmap.put(ASCII.String(url.hash()), url);
added the generation of 50 (!!) new solr field in the core 'webgraph'. The default schema uses only some of them and the resting search index has now the following properties: - webgraph size will have about 40 times as much entries as default index - the complete index size will increase and may be about the double size of current amount As testing showed, not much indexing performance is lost. The default index will be smaller (moved fields out of it); thus searching can be faster. The new index will cause that some old parts in YaCy can be removed, i.e. specialized webgraph data and the noload crawler. The new index will make it possible to: - search within link texts of linked but not indexed documents (about 20 times of document index in size!!) - get a very detailed link graph - enhance ranking using a complete link graph To get the full access to the new index, the API to solr has now two access points: one with attribute core=collection1 for the default search index and core=webgraph to the new webgraph search index. This is also avaiable for p2p operation but client access is not yet implemented.
2013-02-22 15:45:15 +01:00
if (searchEvent != null) {
redesign of index.exist-test: this shall now not be done using a single id to be tested, but with a collection of ids. This will cause only a single call to solr instead of many. The result is a much better performace when testing the existence of many urls. The effect should cause very much less IO during index transmission, both on sender and receiver side.
2013-05-17 13:59:37 +02:00
for (String id: urlmap.keySet()) searchEvent.addHeuristic(ASCII.getBytes(id), heuristicName, true);
}
final Set<String> existing = this.index.exists(urlmap.keySet());
final List<Request> requests = new ArrayList<Request>();
- the webgraph shall store all links which appear on a web page and not all unique links! This made it necessary, that a large portion of the parser and link processing classes must be adopted to carry a different type of link collection which carry a property attribute which are attached to web anchors. - introduction of a new URL class, AnchorURL - the other url classes, DigestURI and MultiProtocolURI had been renamed and refactored to fit into a new document package schema, document.id - cleanup of net.yacy.cora.document package and refactoring
2013-09-15 00:30:23 +02:00
for (Map.Entry<String, DigestURL> e: urlmap.entrySet()) {
redesign of index.exist-test: this shall now not be done using a single id to be tested, but with a collection of ids. This will cause only a single call to solr instead of many. The result is a much better performace when testing the existence of many urls. The effect should cause very much less IO during index transmission, both on sender and receiver side.
2013-05-17 13:59:37 +02:00
final String urlName = e.getValue().toNormalform(true);
if (existing.contains(e.getKey())) {
- refactoring of log to ConcurrentLog: jdk-based logger tend to block at java.util.logging.Logger.log(Logger.java:476) in concurrent environments. This makes logging a main performance issue. To overcome this problem, this is a add-on to jdk logging to put log entries on a concurrent message queue and log the messages one by one using a separate process. - FTPClient uses the concurrent logging instead of the log4j logger
2013-07-09 14:28:25 +02:00
this.log.info("addToIndex: double " + urlName);
redesign of index.exist-test: this shall now not be done using a single id to be tested, but with a collection of ids. This will cause only a single call to solr instead of many. The result is a much better performace when testing the existence of many urls. The effect should cause very much less IO during index transmission, both on sender and receiver side.
2013-05-17 13:59:37 +02:00
continue;
}
final Request request = this.loader.request(e.getValue(), true, true);
self-healing of mistakenly deactivated crawl profiles. This fixes a bug which can happen in rare cases when a crawl start and a cleanup process happen at the same time.
2013-09-25 18:27:54 +02:00
final CrawlProfile profile = this.crawler.get(ASCII.getBytes(request.profileHandle()));
fix for crawl domain counter limitation (limit was reached too early)
2013-09-26 13:41:52 +02:00
final String acceptedError = this.crawlStacker.checkAcceptanceChangeable(e.getValue(), profile, 0);
redesign of index.exist-test: this shall now not be done using a single id to be tested, but with a collection of ids. This will cause only a single call to solr instead of many. The result is a much better performace when testing the existence of many urls. The effect should cause very much less IO during index transmission, both on sender and receiver side.
2013-05-17 13:59:37 +02:00
if (acceptedError != null) {
- refactoring of log to ConcurrentLog: jdk-based logger tend to block at java.util.logging.Logger.log(Logger.java:476) in concurrent environments. This makes logging a main performance issue. To overcome this problem, this is a add-on to jdk logging to put log entries on a concurrent message queue and log the messages one by one using a separate process. - FTPClient uses the concurrent logging instead of the log4j logger
2013-07-09 14:28:25 +02:00
this.log.warn("addToIndex: cannot load " + urlName + ": " + acceptedError);
redesign of index.exist-test: this shall now not be done using a single id to be tested, but with a collection of ids. This will cause only a single call to solr instead of many. The result is a much better performace when testing the existence of many urls. The effect should cause very much less IO during index transmission, both on sender and receiver side.
2013-05-17 13:59:37 +02:00
continue;
}
requests.add(request);
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
}
redesign of index.exist-test: this shall now not be done using a single id to be tested, but with a collection of ids. This will cause only a single call to solr instead of many. The result is a much better performace when testing the existence of many urls. The effect should cause very much less IO during index transmission, both on sender and receiver side.
2013-05-17 13:59:37 +02:00
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
new Thread() {
@Override
public void run() {
redesign of index.exist-test: this shall now not be done using a single id to be tested, but with a collection of ids. This will cause only a single call to solr instead of many. The result is a much better performace when testing the existence of many urls. The effect should cause very much less IO during index transmission, both on sender and receiver side.
2013-05-17 13:59:37 +02:00
for (Request request: requests) {
- the webgraph shall store all links which appear on a web page and not all unique links! This made it necessary, that a large portion of the parser and link processing classes must be adopted to carry a different type of link collection which carry a property attribute which are attached to web anchors. - introduction of a new URL class, AnchorURL - the other url classes, DigestURI and MultiProtocolURI had been renamed and refactored to fit into a new document package schema, document.id - cleanup of net.yacy.cora.document package and refactoring
2013-09-15 00:30:23 +02:00
DigestURL url = request.url();
redesign of index.exist-test: this shall now not be done using a single id to be tested, but with a collection of ids. This will cause only a single call to solr instead of many. The result is a much better performace when testing the existence of many urls. The effect should cause very much less IO during index transmission, both on sender and receiver side.
2013-05-17 13:59:37 +02:00
String urlName = url.toNormalform(true);
Thread.currentThread().setName("Switchboard.addToIndex:" + urlName);
try {
Redesign of crawler identification and robots steering. A non-p2p user in intranets and the internet can now choose to appear as Googlebot. This is an essential necessity to be able to compete in the field of commercial search appliances, since most web pages are these days optimized only for Google and no other search platform any more. All commercial search engine providers have a built-in fake-Google User Agent to be able to get the same search index as Google can do. Without the resistance against obeying to robots.txt in this case, no competition is possible any more. YaCy will always obey the robots.txt when it is used for crawling the web in a peer-to-peer network, but to establish a Search Appliance (like a Google Search Appliance, GSA) it is necessary to be able to behave exactly like a Google crawler. With this change, you will be able to switch the user agent when portal or intranet mode is selected on per-crawl-start basis. Every crawl start can have a different user agent.
2013-08-22 14:23:47 +02:00
final Response response = Switchboard.this.loader.load(request, CacheStrategy.IFFRESH, BlacklistType.CRAWLER, ClientIdentification.yacyIntranetCrawlerAgent);
redesign of index.exist-test: this shall now not be done using a single id to be tested, but with a collection of ids. This will cause only a single call to solr instead of many. The result is a much better performace when testing the existence of many urls. The effect should cause very much less IO during index transmission, both on sender and receiver side.
2013-05-17 13:59:37 +02:00
if (response == null) {
throw new IOException("response == null");
}
if (response.getContent() == null) {
throw new IOException("content == null");
}
if (response.getResponseHeader() == null) {
throw new IOException("header == null");
}
final Document[] documents = response.parse();
if (documents != null) {
for (final Document document: documents) {
self-healing of mistakenly deactivated crawl profiles. This fixes a bug which can happen in rare cases when a crawl start and a cleanup process happen at the same time.
2013-09-25 18:27:54 +02:00
final CrawlProfile profile = crawler.get(ASCII.getBytes(request.profileHandle()));
added a crawl option to obey html-meta-robots-noindex. This is on by default.
2013-07-03 14:50:06 +02:00
if (document.indexingDenied() && (profile == null || profile.obeyHtmlRobotsNoindex())) {
redesign of index.exist-test: this shall now not be done using a single id to be tested, but with a collection of ids. This will cause only a single call to solr instead of many. The result is a much better performace when testing the existence of many urls. The effect should cause very much less IO during index transmission, both on sender and receiver side.
2013-05-17 13:59:37 +02:00
throw new Parser.Failure("indexing is denied", url);
}
final Condenser condenser = new Condenser(document, true, true, LibraryProvider.dymLib, LibraryProvider.synonyms, true);
ResultImages.registerImages(url, document, true);
Switchboard.this.webStructure.generateCitationReference(url, document);
storeDocumentIndex(
response,
collections,
document,
condenser,
searchEvent,
"heuristic:" + heuristicName);
- refactoring of log to ConcurrentLog: jdk-based logger tend to block at java.util.logging.Logger.log(Logger.java:476) in concurrent environments. This makes logging a main performance issue. To overcome this problem, this is a add-on to jdk logging to put log entries on a concurrent message queue and log the messages one by one using a separate process. - FTPClient uses the concurrent logging instead of the log4j logger
2013-07-09 14:28:25 +02:00
Switchboard.this.log.info("addToIndex fill of url " + urlName + " finished");
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
}
}
Added 'final' for all exception blocks as this helps the Java compiler to optimize memory usage Conflicts: source/net/yacy/search/Switchboard.java
2013-07-17 18:31:30 +02:00
} catch (final IOException e ) {
- refactoring of log to ConcurrentLog: jdk-based logger tend to block at java.util.logging.Logger.log(Logger.java:476) in concurrent environments. This makes logging a main performance issue. To overcome this problem, this is a add-on to jdk logging to put log entries on a concurrent message queue and log the messages one by one using a separate process. - FTPClient uses the concurrent logging instead of the log4j logger
2013-07-09 14:28:25 +02:00
Switchboard.this.log.warn("addToIndex: failed loading " + urlName + ": " + e.getMessage());
Added 'final' for all exception blocks as this helps the Java compiler to optimize memory usage Conflicts: source/net/yacy/search/Switchboard.java
2013-07-17 18:31:30 +02:00
} catch (final Parser.Failure e ) {
- refactoring of log to ConcurrentLog: jdk-based logger tend to block at java.util.logging.Logger.log(Logger.java:476) in concurrent environments. This makes logging a main performance issue. To overcome this problem, this is a add-on to jdk logging to put log entries on a concurrent message queue and log the messages one by one using a separate process. - FTPClient uses the concurrent logging instead of the log4j logger
2013-07-09 14:28:25 +02:00
Switchboard.this.log.warn("addToIndex: failed parsing " + urlName + ": " + e.getMessage());
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
}
}
}
}.start();
}
/**
* add url to Crawler - which itself loads the URL, parses the content and adds it to the index
* transparent alternative to "addToIndex" including, double in crawler check, display in crawl monitor
* but doesn't return results for a ongoing search
*
* @param url the url that shall be indexed
* @param asglobal true adds the url to global crawl queue (for remote crawling), false to the local crawler
*/
- the webgraph shall store all links which appear on a web page and not all unique links! This made it necessary, that a large portion of the parser and link processing classes must be adopted to carry a different type of link collection which carry a property attribute which are attached to web anchors. - introduction of a new URL class, AnchorURL - the other url classes, DigestURI and MultiProtocolURI had been renamed and refactored to fit into a new document package schema, document.id - cleanup of net.yacy.cora.document package and refactoring
2013-09-15 00:30:23 +02:00
public void addToCrawler(final Collection<DigestURL> urls, final boolean asglobal) {
Map<String, DigestURL> urlmap = new HashMap<String, DigestURL>();
for (DigestURL url: urls) urlmap.put(ASCII.String(url.hash()), url);
redesign of index.exist-test: this shall now not be done using a single id to be tested, but with a collection of ids. This will cause only a single call to solr instead of many. The result is a much better performace when testing the existence of many urls. The effect should cause very much less IO during index transmission, both on sender and receiver side.
2013-05-17 13:59:37 +02:00
Set<String> existingids = this.index.exists(urlmap.keySet());
- the webgraph shall store all links which appear on a web page and not all unique links! This made it necessary, that a large portion of the parser and link processing classes must be adopted to carry a different type of link collection which carry a property attribute which are attached to web anchors. - introduction of a new URL class, AnchorURL - the other url classes, DigestURI and MultiProtocolURI had been renamed and refactored to fit into a new document package schema, document.id - cleanup of net.yacy.cora.document package and refactoring
2013-09-15 00:30:23 +02:00
for (Map.Entry<String, DigestURL> e: urlmap.entrySet()) {
redesign of index.exist-test: this shall now not be done using a single id to be tested, but with a collection of ids. This will cause only a single call to solr instead of many. The result is a much better performace when testing the existence of many urls. The effect should cause very much less IO during index transmission, both on sender and receiver side.
2013-05-17 13:59:37 +02:00
if (existingids.contains(e.getKey())) continue; // double
- the webgraph shall store all links which appear on a web page and not all unique links! This made it necessary, that a large portion of the parser and link processing classes must be adopted to carry a different type of link collection which carry a property attribute which are attached to web anchors. - introduction of a new URL class, AnchorURL - the other url classes, DigestURI and MultiProtocolURI had been renamed and refactored to fit into a new document package schema, document.id - cleanup of net.yacy.cora.document package and refactoring
2013-09-15 00:30:23 +02:00
DigestURL url = e.getValue();
redesign of index.exist-test: this shall now not be done using a single id to be tested, but with a collection of ids. This will cause only a single call to solr instead of many. The result is a much better performace when testing the existence of many urls. The effect should cause very much less IO during index transmission, both on sender and receiver side.
2013-05-17 13:59:37 +02:00
final Request request = this.loader.request(url, true, true);
self-healing of mistakenly deactivated crawl profiles. This fixes a bug which can happen in rare cases when a crawl start and a cleanup process happen at the same time.
2013-09-25 18:27:54 +02:00
final CrawlProfile profile = this.crawler.get(ASCII.getBytes(request.profileHandle()));
fix for crawl domain counter limitation (limit was reached too early)
2013-09-26 13:41:52 +02:00
String acceptedError = this.crawlStacker.checkAcceptanceChangeable(url, profile, 0);
if (acceptedError == null) acceptedError = this.crawlStacker.checkAcceptanceInitially(url, profile);
redesign of index.exist-test: this shall now not be done using a single id to be tested, but with a collection of ids. This will cause only a single call to solr instead of many. The result is a much better performace when testing the existence of many urls. The effect should cause very much less IO during index transmission, both on sender and receiver side.
2013-05-17 13:59:37 +02:00
if (acceptedError != null) {
- refactoring of log to ConcurrentLog: jdk-based logger tend to block at java.util.logging.Logger.log(Logger.java:476) in concurrent environments. This makes logging a main performance issue. To overcome this problem, this is a add-on to jdk logging to put log entries on a concurrent message queue and log the messages one by one using a separate process. - FTPClient uses the concurrent logging instead of the log4j logger
2013-07-09 14:28:25 +02:00
this.log.info("addToCrawler: cannot load " + url.toNormalform(true) + ": " + acceptedError);
redesign of index.exist-test: this shall now not be done using a single id to be tested, but with a collection of ids. This will cause only a single call to solr instead of many. The result is a much better performace when testing the existence of many urls. The effect should cause very much less IO during index transmission, both on sender and receiver side.
2013-05-17 13:59:37 +02:00
return;
}
final String s;
if (asglobal) {
fix for crawl stack domain counter
2013-07-16 18:18:55 +02:00
s = this.crawlQueues.noticeURL.push(StackType.GLOBAL, request, profile, this.robots);
redesign of index.exist-test: this shall now not be done using a single id to be tested, but with a collection of ids. This will cause only a single call to solr instead of many. The result is a much better performace when testing the existence of many urls. The effect should cause very much less IO during index transmission, both on sender and receiver side.
2013-05-17 13:59:37 +02:00
} else {
fix for crawl stack domain counter
2013-07-16 18:18:55 +02:00
s = this.crawlQueues.noticeURL.push(StackType.LOCAL, request, profile, this.robots);
redesign of index.exist-test: this shall now not be done using a single id to be tested, but with a collection of ids. This will cause only a single call to solr instead of many. The result is a much better performace when testing the existence of many urls. The effect should cause very much less IO during index transmission, both on sender and receiver side.
2013-05-17 13:59:37 +02:00
}
if (s != null) {
- refactoring of log to ConcurrentLog: jdk-based logger tend to block at java.util.logging.Logger.log(Logger.java:476) in concurrent environments. This makes logging a main performance issue. To overcome this problem, this is a add-on to jdk logging to put log entries on a concurrent message queue and log the messages one by one using a separate process. - FTPClient uses the concurrent logging instead of the log4j logger
2013-07-09 14:28:25 +02:00
Switchboard.this.log.info("addToCrawler: failed to add " + url.toNormalform(true) + ": " + s);
redesign of index.exist-test: this shall now not be done using a single id to be tested, but with a collection of ids. This will cause only a single call to solr instead of many. The result is a much better performace when testing the existence of many urls. The effect should cause very much less IO during index transmission, both on sender and receiver side.
2013-05-17 13:59:37 +02:00
}
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
}
}
public class receiptSending implements Runnable
{
private final Seed initiatorPeer;
private final URIMetadataNode reference;
public receiptSending(final Seed initiatorPeer, final URIMetadataNode reference) {
this.initiatorPeer = initiatorPeer;
this.reference = reference;
}
@Override
public void run() {
final long t = System.currentTimeMillis();
final Map<String, String> response =
Protocol.crawlReceipt(
Switchboard.this.peers.mySeed(),
this.initiatorPeer,
"crawl",
"fill",
"indexed",
this.reference,
"");
if ( response == null ) {
- refactoring of log to ConcurrentLog: jdk-based logger tend to block at java.util.logging.Logger.log(Logger.java:476) in concurrent environments. This makes logging a main performance issue. To overcome this problem, this is a add-on to jdk logging to put log entries on a concurrent message queue and log the messages one by one using a separate process. - FTPClient uses the concurrent logging instead of the log4j logger
2013-07-09 14:28:25 +02:00
Switchboard.this.log.info("Sending crawl receipt for '"
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
+ this.reference.url().toNormalform(true)
+ "' to "
+ this.initiatorPeer.getName()
+ " FAILED, send time = "
+ (System.currentTimeMillis() - t));
return;
}
final String delay = response.get("delay");
- refactoring of log to ConcurrentLog: jdk-based logger tend to block at java.util.logging.Logger.log(Logger.java:476) in concurrent environments. This makes logging a main performance issue. To overcome this problem, this is a add-on to jdk logging to put log entries on a concurrent message queue and log the messages one by one using a separate process. - FTPClient uses the concurrent logging instead of the log4j logger
2013-07-09 14:28:25 +02:00
Switchboard.this.log.info("Sending crawl receipt for '"
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
+ this.reference.url().toNormalform(true)
+ "' to "
+ this.initiatorPeer.getName()
+ " success, delay = "
+ delay
+ ", send time = "
+ (System.currentTimeMillis() - t));
}
}
automatically running optimize on solr if user/search is idle for some time
2013-07-14 10:02:08 +02:00
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
/**
* check authentication status for request access shall be granted if return value >= 2; these are the
* cases where an access is granted to protected pages: - a password is not configured: auth-level 2 -
* access from localhost is granted and access comes from localhost: auth-level 3 - a password is
* configured and access comes from localhost and the realm-value of a http-authentify String is equal to
* the stored base64MD5: auth-level 3 - a password is configured and access comes with matching
* http-authentify: auth-level 4
*
* @param requestHeader
* @return the auth-level as described above or 1 which means 'not authorized'. a 0 is returned in case of
* fraud attempts
*/
public int adminAuthenticated(final RequestHeader requestHeader) {
// authorization in case that there is no account stored
fixed shutdown authenticaton problem
2014-01-06 01:48:54 +01:00
final String adminAccountUserName = getConfig(SwitchboardConstants.ADMIN_ACCOUNT_USER_NAME, "admin");
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
final String adminAccountBase64MD5 = getConfig(SwitchboardConstants.ADMIN_ACCOUNT_B64MD5, "");
if ( adminAccountBase64MD5.isEmpty() ) {
automatically running optimize on solr if user/search is idle for some time
2013-07-14 10:02:08 +02:00
adminAuthenticationLastAccess = System.currentTimeMillis();
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
return 2; // no password stored; this should not happen for older peers
}
// authorization for localhost, only if flag is set to grant localhost access as admin
final boolean accessFromLocalhost = requestHeader.accessFromLocalhost();
refactoring (usage of constant names for attributes of authentication check)
2014-01-05 04:23:44 +01:00
if ( getConfigBool(SwitchboardConstants.ADMIN_ACCOUNT_FOR_LOCALHOST, false) && accessFromLocalhost ) {
automatically running optimize on solr if user/search is idle for some time
2013-07-14 10:02:08 +02:00
adminAuthenticationLastAccess = System.currentTimeMillis();
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
return 3; // soft-authenticated for localhost
}
// get the authorization string from the header
final String realmProp = (requestHeader.get(RequestHeader.AUTHORIZATION, "xxxxxx")).trim();
final String realmValue = realmProp.substring(6);
// security check against too long authorization strings
if ( realmValue.length() > 256 ) {
return 0;
}
// authorization by encoded password, only for localhost access
fixed shutdown authenticaton problem
2014-01-06 01:48:54 +01:00
String pass = Base64Order.standardCoder.encodeString(adminAccountUserName + ":" + adminAccountBase64MD5);
if ( accessFromLocalhost && (pass.equals(realmValue)) ) {
automatically running optimize on solr if user/search is idle for some time
2013-07-14 10:02:08 +02:00
adminAuthenticationLastAccess = System.currentTimeMillis();
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
return 3; // soft-authenticated for localhost
}
// authorization by hit in userDB
if ( this.userDB.hasAdminRight(realmProp, requestHeader.getHeaderCookies()) ) {
automatically running optimize on solr if user/search is idle for some time
2013-07-14 10:02:08 +02:00
adminAuthenticationLastAccess = System.currentTimeMillis();
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
return 4; //return, because 4=max
}
// authorization with admin keyword in configuration
if ( realmValue == null || realmValue.isEmpty() ) {
return 1;
}
if ( adminAccountBase64MD5.equals(Digest.encodeMD5Hex(realmValue)) ) {
automatically running optimize on solr if user/search is idle for some time
2013-07-14 10:02:08 +02:00
adminAuthenticationLastAccess = System.currentTimeMillis();
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
return 4; // hard-authenticated, all ok
}
return 1;
}
public boolean verifyAuthentication(final RequestHeader header) {
// handle access rights
switch ( adminAuthenticated(header) ) {
case 0: // wrong password given
//try { Thread.sleep(3000); } catch (final InterruptedException e) { } // prevent brute-force
return false;
case 1: // no password given
return false;
case 2: // no password stored
return true;
case 3: // soft-authenticated for localhost only
return true;
case 4: // hard-authenticated, all ok
return true;
default:
return false;
}
}
public void setPerformance(final int wantedPPM) {
int wPPM = wantedPPM;
// we consider 3 cases here
// wantedPPM <= 10: low performance
// 10 < wantedPPM < 30000: custom performance
// 30000 <= wantedPPM : maximum performance
if ( wPPM <= 0 ) {
wPPM = 1;
}
if ( wPPM >= 30000 ) {
wPPM = 30000;
}
final int newBusySleep = 60000 / wPPM; // for wantedPPM = 10: 6000; for wantedPPM = 1000: 60
BusyThread thread;
thread = getThread(SwitchboardConstants.INDEX_DIST);
if ( thread != null ) {
setConfig(
SwitchboardConstants.INDEX_DIST_BUSYSLEEP,
thread.setBusySleep(Math.max(2000, thread.setBusySleep(newBusySleep * 2))));
thread.setIdleSleep(30000);
}
thread = getThread(SwitchboardConstants.CRAWLJOB_LOCAL_CRAWL);
if ( thread != null ) {
setConfig(SwitchboardConstants.CRAWLJOB_LOCAL_CRAWL_BUSYSLEEP, thread.setBusySleep(newBusySleep));
thread.setIdleSleep(2000);
}
}
public String dhtShallTransfer() {
final String cautionCause = onlineCaution();
if ( cautionCause != null ) {
return "online caution for " + cautionCause + ", dht transmission";
}
if ( this.peers == null ) {
return "no DHT distribution: seedDB == null";
}
if ( this.peers.mySeed() == null ) {
return "no DHT distribution: mySeed == null";
}
if ( this.peers.mySeed().isVirgin() ) {
return "no DHT distribution: status is virgin";
}
if ( this.peers.noDHTActivity() ) {
return "no DHT distribution: network too small";
}
if ( !getConfigBool(SwitchboardConstants.DHT_ENABLED, true) ) {
return "no DHT distribution: disabled by network.unit.dht";
}
if ( getConfig(SwitchboardConstants.INDEX_DIST_ALLOW, "false").equalsIgnoreCase("false") ) {
return "no DHT distribution: not enabled (per setting)";
}
final Segment indexSegment = this.index;
if ( indexSegment.RWICount() < 100 ) {
return "no DHT distribution: not enough words - wordIndex.size() = "
+ indexSegment.RWICount();
}
removed size request
2013-02-26 17:53:44 +01:00
if ( (getConfig(SwitchboardConstants.INDEX_DIST_ALLOW_WHILE_CRAWLING, "false").equalsIgnoreCase("false")) && (this.crawlQueues.noticeURL.notEmptyLocal()) ) {
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
return "no DHT distribution: crawl in progress: noticeURL.stackSize() = "
+ this.crawlQueues.noticeURL.size()
+ ", sbQueue.size() = "
+ getIndexingProcessorsQueueSize();
}
removed size request
2013-02-26 17:53:44 +01:00
if ( (getConfig(SwitchboardConstants.INDEX_DIST_ALLOW_WHILE_INDEXING, "false").equalsIgnoreCase("false")) && (getIndexingProcessorsQueueSize() > 1) ) {
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
return "no DHT distribution: indexing in progress: noticeURL.stackSize() = "
+ this.crawlQueues.noticeURL.size()
+ ", sbQueue.size() = "
+ getIndexingProcessorsQueueSize();
}
do not run dht ditribution if system load is over 2.5
2013-07-23 19:32:32 +02:00
double load = Memory.load();
float maxload = getConfigFloat(SwitchboardConstants.INDEX_TRANSFER_MAXLOAD, 1.5f);
if (load > maxload) {
return "no DHT distribution: system has too high load = " + load + ", maxload = " + maxload;
}
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
return null; // this means; yes, please do dht transfer
}
public boolean dhtTransferJob() {
if ( this.dhtDispatcher == null ) {
return false;
}
final String rejectReason = dhtShallTransfer();
if ( rejectReason != null ) {
if ( this.log.isFine() ) {
- refactoring of log to ConcurrentLog: jdk-based logger tend to block at java.util.logging.Logger.log(Logger.java:476) in concurrent environments. This makes logging a main performance issue. To overcome this problem, this is a add-on to jdk logging to put log entries on a concurrent message queue and log the messages one by one using a separate process. - FTPClient uses the concurrent logging instead of the log4j logger
2013-07-09 14:28:25 +02:00
this.log.fine(rejectReason);
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
}
return false;
}
boolean hasDoneSomething = false;
final long kbytesUp = ConnectionInfo.getActiveUpbytes() / 1024;
// accumulate RWIs to transmission cloud
if ( this.dhtDispatcher.cloudSize() > this.peers.scheme.verticalPartitions() ) {
- refactoring of log to ConcurrentLog: jdk-based logger tend to block at java.util.logging.Logger.log(Logger.java:476) in concurrent environments. This makes logging a main performance issue. To overcome this problem, this is a add-on to jdk logging to put log entries on a concurrent message queue and log the messages one by one using a separate process. - FTPClient uses the concurrent logging instead of the log4j logger
2013-07-09 14:28:25 +02:00
this.log.info("dhtTransferJob: no selection, too many entries in transmission cloud: "
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
+ this.dhtDispatcher.cloudSize());
} else if ( MemoryControl.available() < 1024 * 1024 * 25 ) {
- refactoring of log to ConcurrentLog: jdk-based logger tend to block at java.util.logging.Logger.log(Logger.java:476) in concurrent environments. This makes logging a main performance issue. To overcome this problem, this is a add-on to jdk logging to put log entries on a concurrent message queue and log the messages one by one using a separate process. - FTPClient uses the concurrent logging instead of the log4j logger
2013-07-09 14:28:25 +02:00
this.log.info("dhtTransferJob: no selection, too less memory available : "
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
+ (MemoryControl.available() / 1024 / 1024)
+ " MB");
} else if ( ConnectionInfo.getLoadPercent() > 50 ) {
- refactoring of log to ConcurrentLog: jdk-based logger tend to block at java.util.logging.Logger.log(Logger.java:476) in concurrent environments. This makes logging a main performance issue. To overcome this problem, this is a add-on to jdk logging to put log entries on a concurrent message queue and log the messages one by one using a separate process. - FTPClient uses the concurrent logging instead of the log4j logger
2013-07-09 14:28:25 +02:00
this.log.info("dhtTransferJob: too many connections in httpc pool : "
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
+ ConnectionInfo.getCount());
// close unused connections
// Client.cleanup();
} else if ( kbytesUp > 128 ) {
- refactoring of log to ConcurrentLog: jdk-based logger tend to block at java.util.logging.Logger.log(Logger.java:476) in concurrent environments. This makes logging a main performance issue. To overcome this problem, this is a add-on to jdk logging to put log entries on a concurrent message queue and log the messages one by one using a separate process. - FTPClient uses the concurrent logging instead of the log4j logger
2013-07-09 14:28:25 +02:00
this.log.info("dhtTransferJob: too much upload(1), currently uploading: " + kbytesUp + " Kb");
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
} else {
byte[] startHash = null, limitHash = null;
int tries = 10;
while ( tries-- > 0 ) {
startHash = DHTSelection.selectTransferStart();
assert startHash != null;
limitHash = DHTSelection.limitOver(this.peers, startHash);
if ( limitHash != null ) {
break;
}
}
if ( limitHash == null || startHash == null ) {
- refactoring of log to ConcurrentLog: jdk-based logger tend to block at java.util.logging.Logger.log(Logger.java:476) in concurrent environments. This makes logging a main performance issue. To overcome this problem, this is a add-on to jdk logging to put log entries on a concurrent message queue and log the messages one by one using a separate process. - FTPClient uses the concurrent logging instead of the log4j logger
2013-07-09 14:28:25 +02:00
this.log.info("dhtTransferJob: approaching full DHT dispersion.");
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
return false;
}
- refactoring of log to ConcurrentLog: jdk-based logger tend to block at java.util.logging.Logger.log(Logger.java:476) in concurrent environments. This makes logging a main performance issue. To overcome this problem, this is a add-on to jdk logging to put log entries on a concurrent message queue and log the messages one by one using a separate process. - FTPClient uses the concurrent logging instead of the log4j logger
2013-07-09 14:28:25 +02:00
this.log.info("dhtTransferJob: selected " + ASCII.String(startHash) + " as start hash");
this.log.info("dhtTransferJob: selected " + ASCII.String(limitHash) + " as limit hash");
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
final boolean enqueued =
this.dhtDispatcher.selectContainersEnqueueToCloud(
startHash,
limitHash,
dhtMaxContainerCount,
this.dhtMaxReferenceCount,
5000);
hasDoneSomething = hasDoneSomething | enqueued;
- refactoring of log to ConcurrentLog: jdk-based logger tend to block at java.util.logging.Logger.log(Logger.java:476) in concurrent environments. This makes logging a main performance issue. To overcome this problem, this is a add-on to jdk logging to put log entries on a concurrent message queue and log the messages one by one using a separate process. - FTPClient uses the concurrent logging instead of the log4j logger
2013-07-09 14:28:25 +02:00
this.log.info("dhtTransferJob: result from enqueueing: " + ((enqueued) ? "true" : "false"));
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
}
// check if we can deliver entries to other peers
if ( this.dhtDispatcher.transmissionSize() >= 10 ) {
this.log
- refactoring of log to ConcurrentLog: jdk-based logger tend to block at java.util.logging.Logger.log(Logger.java:476) in concurrent environments. This makes logging a main performance issue. To overcome this problem, this is a add-on to jdk logging to put log entries on a concurrent message queue and log the messages one by one using a separate process. - FTPClient uses the concurrent logging instead of the log4j logger
2013-07-09 14:28:25 +02:00
.info("dhtTransferJob: no dequeueing from cloud to transmission: too many concurrent sessions: "
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
+ this.dhtDispatcher.transmissionSize());
} else if ( ConnectionInfo.getLoadPercent() > 75 ) {
- refactoring of log to ConcurrentLog: jdk-based logger tend to block at java.util.logging.Logger.log(Logger.java:476) in concurrent environments. This makes logging a main performance issue. To overcome this problem, this is a add-on to jdk logging to put log entries on a concurrent message queue and log the messages one by one using a separate process. - FTPClient uses the concurrent logging instead of the log4j logger
2013-07-09 14:28:25 +02:00
this.log.info("dhtTransferJob: too many connections in httpc pool : "
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
+ ConnectionInfo.getCount());
// close unused connections
// Client.cleanup();
} else if ( kbytesUp > 256 ) {
- refactoring of log to ConcurrentLog: jdk-based logger tend to block at java.util.logging.Logger.log(Logger.java:476) in concurrent environments. This makes logging a main performance issue. To overcome this problem, this is a add-on to jdk logging to put log entries on a concurrent message queue and log the messages one by one using a separate process. - FTPClient uses the concurrent logging instead of the log4j logger
2013-07-09 14:28:25 +02:00
this.log.info("dhtTransferJob: too much upload(2), currently uploading: " + kbytesUp + " Kb");
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
} else {
final boolean dequeued = this.dhtDispatcher.dequeueContainer();
hasDoneSomething = hasDoneSomething | dequeued;
- refactoring of log to ConcurrentLog: jdk-based logger tend to block at java.util.logging.Logger.log(Logger.java:476) in concurrent environments. This makes logging a main performance issue. To overcome this problem, this is a add-on to jdk logging to put log entries on a concurrent message queue and log the messages one by one using a separate process. - FTPClient uses the concurrent logging instead of the log4j logger
2013-07-09 14:28:25 +02:00
this.log.info("dhtTransferJob: result from dequeueing: " + ((dequeued) ? "true" : "false"));
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
}
return hasDoneSomething;
}
public final void heuristicSite(final SearchEvent searchEvent, final String host) {
new Thread() {
@Override
public void run() {
Thread.currentThread().setName("Switchboard.heuristicSite:" + host);
String r = host;
if ( r.indexOf("//", 0) < 0 ) {
r = "http://" + r;
}
// get the links for a specific site
- replaced the properties object in AnchorURL with distinct variables for anchor attributes. - this caused that large portions of the parser code had to be adopted as well - added a counter target_order_i for anchor links in webgraph computation
2013-09-15 23:27:04 +02:00
AnchorURL url;
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
try {
- replaced the properties object in AnchorURL with distinct variables for anchor attributes. - this caused that large portions of the parser code had to be adopted as well - added a counter target_order_i for anchor links in webgraph computation
2013-09-15 23:27:04 +02:00
url = new AnchorURL(r);
Added 'final' for all exception blocks as this helps the Java compiler to optimize memory usage Conflicts: source/net/yacy/search/Switchboard.java
2013-07-17 18:31:30 +02:00
} catch (final MalformedURLException e ) {
- refactoring of log to ConcurrentLog: jdk-based logger tend to block at java.util.logging.Logger.log(Logger.java:476) in concurrent environments. This makes logging a main performance issue. To overcome this problem, this is a add-on to jdk logging to put log entries on a concurrent message queue and log the messages one by one using a separate process. - FTPClient uses the concurrent logging instead of the log4j logger
2013-07-09 14:28:25 +02:00
ConcurrentLog.logException(e);
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
return;
}
- the webgraph shall store all links which appear on a web page and not all unique links! This made it necessary, that a large portion of the parser and link processing classes must be adopted to carry a different type of link collection which carry a property attribute which are attached to web anchors. - introduction of a new URL class, AnchorURL - the other url classes, DigestURI and MultiProtocolURI had been renamed and refactored to fit into a new document package schema, document.id - cleanup of net.yacy.cora.document package and refactoring
2013-09-15 00:30:23 +02:00
final Map<DigestURL, String> links;
complete redesign of search process: - removed 'worker' processes - no internal time-out behaviour: methods either are successful or return null - waiting is only done on top-level - removed snippet-production; this is replaced by solr snippets - removed statistics based on solr size queries (they had been VERY long); the statistics (like suggestions or tag cloud) are now again based on the old but very fast RWI index. In portal or intranet mode the RWI index is usually switched off; if you like to have statistics again then you must switch on the rwis again in this mode. - fixed many bugs regarding correct page counter
2013-02-26 17:16:31 +01:00
searchEvent.oneFeederStarted();
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
try {
Redesign of crawler identification and robots steering. A non-p2p user in intranets and the internet can now choose to appear as Googlebot. This is an essential necessity to be able to compete in the field of commercial search appliances, since most web pages are these days optimized only for Google and no other search platform any more. All commercial search engine providers have a built-in fake-Google User Agent to be able to get the same search index as Google can do. Without the resistance against obeying to robots.txt in this case, no competition is possible any more. YaCy will always obey the robots.txt when it is used for crawling the web in a peer-to-peer network, but to establish a Search Appliance (like a Google Search Appliance, GSA) it is necessary to be able to behave exactly like a Google crawler. With this change, you will be able to switch the user agent when portal or intranet mode is selected on per-crawl-start basis. Every crawl start can have a different user agent.
2013-08-22 14:23:47 +02:00
links = Switchboard.this.loader.loadLinks(url, CacheStrategy.NOCACHE, BlacklistType.SEARCH, ClientIdentification.yacyIntranetCrawlerAgent);
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
if ( links != null ) {
- the webgraph shall store all links which appear on a web page and not all unique links! This made it necessary, that a large portion of the parser and link processing classes must be adopted to carry a different type of link collection which carry a property attribute which are attached to web anchors. - introduction of a new URL class, AnchorURL - the other url classes, DigestURI and MultiProtocolURI had been renamed and refactored to fit into a new document package schema, document.id - cleanup of net.yacy.cora.document package and refactoring
2013-09-15 00:30:23 +02:00
final Iterator<DigestURL> i = links.keySet().iterator();
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
while ( i.hasNext() ) {
if ( !i.next().getHost().endsWith(host) ) {
i.remove();
}
}
// add all pages to the index
added a feature to set a collection for a crawl result based on a regular expression on th url: the collection attribut for a crawl start may be now either a token or a list of tokens, seperated by ',' where a token is either a string or a pair <string,pattern> where the string is separated to the pattern with a ':' and the string is assigned to the document as collection only if the pattern matches with the url.
2013-08-25 00:13:48 +02:00
addAllToIndex(url, links, searchEvent, "site", CrawlProfile.collectionParser("site"));
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
}
Added 'final' for all exception blocks as this helps the Java compiler to optimize memory usage Conflicts: source/net/yacy/search/Switchboard.java
2013-07-17 18:31:30 +02:00
} catch (final Throwable e ) {
- refactoring of log to ConcurrentLog: jdk-based logger tend to block at java.util.logging.Logger.log(Logger.java:476) in concurrent environments. This makes logging a main performance issue. To overcome this problem, this is a add-on to jdk logging to put log entries on a concurrent message queue and log the messages one by one using a separate process. - FTPClient uses the concurrent logging instead of the log4j logger
2013-07-09 14:28:25 +02:00
ConcurrentLog.logException(e);
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
} finally {
complete redesign of search process: - removed 'worker' processes - no internal time-out behaviour: methods either are successful or return null - waiting is only done on top-level - removed snippet-production; this is replaced by solr snippets - removed statistics based on solr size queries (they had been VERY long); the statistics (like suggestions or tag cloud) are now again based on the old but very fast RWI index. In portal or intranet mode the RWI index is usually switched off; if you like to have statistics again then you must switch on the rwis again in this mode. - fixed many bugs regarding correct page counter
2013-02-26 17:16:31 +01:00
searchEvent.oneFeederTerminated();
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
}
}
}.start();
}
added a 'greedy learning' mechanismn which will cause that a 'fresh' yacy will load linked web pages from search results until the total number of web pages reaches 15000. This shall give fresh peers a 'boost' to get faster a personalized search index.
2013-06-11 14:42:30 +02:00
public final void heuristicSearchResults(final String url) {
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
new Thread() {
@Override
public void run() {
// get the links for a specific site
- replaced the properties object in AnchorURL with distinct variables for anchor attributes. - this caused that large portions of the parser code had to be adopted as well - added a counter target_order_i for anchor links in webgraph computation
2013-09-15 23:27:04 +02:00
final AnchorURL startUrl;
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
try {
- replaced the properties object in AnchorURL with distinct variables for anchor attributes. - this caused that large portions of the parser code had to be adopted as well - added a counter target_order_i for anchor links in webgraph computation
2013-09-15 23:27:04 +02:00
startUrl = new AnchorURL(url);
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
} catch (final MalformedURLException e) {
- refactoring of log to ConcurrentLog: jdk-based logger tend to block at java.util.logging.Logger.log(Logger.java:476) in concurrent environments. This makes logging a main performance issue. To overcome this problem, this is a add-on to jdk logging to put log entries on a concurrent message queue and log the messages one by one using a separate process. - FTPClient uses the concurrent logging instead of the log4j logger
2013-07-09 14:28:25 +02:00
ConcurrentLog.logException(e);
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
return;
}
- the webgraph shall store all links which appear on a web page and not all unique links! This made it necessary, that a large portion of the parser and link processing classes must be adopted to carry a different type of link collection which carry a property attribute which are attached to web anchors. - introduction of a new URL class, AnchorURL - the other url classes, DigestURI and MultiProtocolURI had been renamed and refactored to fit into a new document package schema, document.id - cleanup of net.yacy.cora.document package and refactoring
2013-09-15 00:30:23 +02:00
final Map<DigestURL, String> links;
DigestURL url;
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
try {
Redesign of crawler identification and robots steering. A non-p2p user in intranets and the internet can now choose to appear as Googlebot. This is an essential necessity to be able to compete in the field of commercial search appliances, since most web pages are these days optimized only for Google and no other search platform any more. All commercial search engine providers have a built-in fake-Google User Agent to be able to get the same search index as Google can do. Without the resistance against obeying to robots.txt in this case, no competition is possible any more. YaCy will always obey the robots.txt when it is used for crawling the web in a peer-to-peer network, but to establish a Search Appliance (like a Google Search Appliance, GSA) it is necessary to be able to behave exactly like a Google crawler. With this change, you will be able to switch the user agent when portal or intranet mode is selected on per-crawl-start basis. Every crawl start can have a different user agent.
2013-08-22 14:23:47 +02:00
links = Switchboard.this.loader.loadLinks(startUrl, CacheStrategy.IFFRESH, BlacklistType.SEARCH, ClientIdentification.yacyIntranetCrawlerAgent);
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
if (links != null) {
if (links.size() < 1000) { // limit to 1000 to skip large index pages
- the webgraph shall store all links which appear on a web page and not all unique links! This made it necessary, that a large portion of the parser and link processing classes must be adopted to carry a different type of link collection which carry a property attribute which are attached to web anchors. - introduction of a new URL class, AnchorURL - the other url classes, DigestURI and MultiProtocolURI had been renamed and refactored to fit into a new document package schema, document.id - cleanup of net.yacy.cora.document package and refactoring
2013-09-15 00:30:23 +02:00
final Iterator<DigestURL> i = links.keySet().iterator();
added a 'greedy learning' mechanismn which will cause that a 'fresh' yacy will load linked web pages from search results until the total number of web pages reaches 15000. This shall give fresh peers a 'boost' to get faster a personalized search index.
2013-06-11 14:42:30 +02:00
final boolean globalcrawljob = Switchboard.this.getConfigBool(SwitchboardConstants.HEURISTIC_SEARCHRESULTS_CRAWLGLOBAL,false);
- the webgraph shall store all links which appear on a web page and not all unique links! This made it necessary, that a large portion of the parser and link processing classes must be adopted to carry a different type of link collection which carry a property attribute which are attached to web anchors. - introduction of a new URL class, AnchorURL - the other url classes, DigestURI and MultiProtocolURI had been renamed and refactored to fit into a new document package schema, document.id - cleanup of net.yacy.cora.document package and refactoring
2013-09-15 00:30:23 +02:00
Collection<DigestURL> urls = new ArrayList<DigestURL>();
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
while (i.hasNext()) {
added the generation of 50 (!!) new solr field in the core 'webgraph'. The default schema uses only some of them and the resting search index has now the following properties: - webgraph size will have about 40 times as much entries as default index - the complete index size will increase and may be about the double size of current amount As testing showed, not much indexing performance is lost. The default index will be smaller (moved fields out of it); thus searching can be faster. The new index will cause that some old parts in YaCy can be removed, i.e. specialized webgraph data and the noload crawler. The new index will make it possible to: - search within link texts of linked but not indexed documents (about 20 times of document index in size!!) - get a very detailed link graph - enhance ranking using a complete link graph To get the full access to the new index, the API to solr has now two access points: one with attribute core=collection1 for the default search index and core=webgraph to the new webgraph search index. This is also avaiable for p2p operation but client access is not yet implemented.
2013-02-22 15:45:15 +01:00
url = i.next();
fix for host compare in case that the host is null. This happens when doing a search in the intranet for file resources (they don't have a host).
2013-06-10 16:23:58 +02:00
boolean islocal = (url.getHost() == null && startUrl.getHost() == null) || (url.getHost() != null && startUrl.getHost() != null && url.getHost().contentEquals(startUrl.getHost()));
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
// add all external links or links to different page to crawler
if ( !islocal ) {// || (!startUrl.getPath().endsWith(url.getPath()))) {
redesign of index.exist-test: this shall now not be done using a single id to be tested, but with a collection of ids. This will cause only a single call to solr instead of many. The result is a much better performace when testing the existence of many urls. The effect should cause very much less IO during index transmission, both on sender and receiver side.
2013-05-17 13:59:37 +02:00
urls.add(url);
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
}
}
redesign of index.exist-test: this shall now not be done using a single id to be tested, but with a collection of ids. This will cause only a single call to solr instead of many. The result is a much better performace when testing the existence of many urls. The effect should cause very much less IO during index transmission, both on sender and receiver side.
2013-05-17 13:59:37 +02:00
addToCrawler(urls, globalcrawljob);
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
}
}
} catch (final Throwable e) {
}
}
}.start();
}
// blekko pattern: http://blekko.com/ws/$+/rss
public final void heuristicRSS(
final String urlpattern,
final SearchEvent searchEvent,
final String feedName) {
final int p = urlpattern.indexOf('$');
if ( p < 0 ) {
return;
}
new Thread() {
@Override
public void run() {
String queryString = searchEvent.query.getQueryGoal().getOriginalQueryString(false);
Thread.currentThread().setName("Switchboard.heuristicRSS:" + queryString);
final int meta = queryString.indexOf("heuristic:", 0);
if ( meta >= 0 ) {
final int q = queryString.indexOf(' ', meta);
if ( q >= 0 ) {
queryString = queryString.substring(0, meta) + queryString.substring(q + 1);
} else {
queryString = queryString.substring(0, meta);
}
}
final String urlString =
urlpattern.substring(0, p)
+ queryString.trim().replaceAll(" ", "+")
+ urlpattern.substring(p + 1);
- the webgraph shall store all links which appear on a web page and not all unique links! This made it necessary, that a large portion of the parser and link processing classes must be adopted to carry a different type of link collection which carry a property attribute which are attached to web anchors. - introduction of a new URL class, AnchorURL - the other url classes, DigestURI and MultiProtocolURI had been renamed and refactored to fit into a new document package schema, document.id - cleanup of net.yacy.cora.document package and refactoring
2013-09-15 00:30:23 +02:00
final DigestURL url;
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
try {
- the webgraph shall store all links which appear on a web page and not all unique links! This made it necessary, that a large portion of the parser and link processing classes must be adopted to carry a different type of link collection which carry a property attribute which are attached to web anchors. - introduction of a new URL class, AnchorURL - the other url classes, DigestURI and MultiProtocolURI had been renamed and refactored to fit into a new document package schema, document.id - cleanup of net.yacy.cora.document package and refactoring
2013-09-15 00:30:23 +02:00
url = new DigestURL(MultiProtocolURL.unescape(urlString));
Added 'final' for all exception blocks as this helps the Java compiler to optimize memory usage Conflicts: source/net/yacy/search/Switchboard.java
2013-07-17 18:31:30 +02:00
} catch (final MalformedURLException e1 ) {
- refactoring of log to ConcurrentLog: jdk-based logger tend to block at java.util.logging.Logger.log(Logger.java:476) in concurrent environments. This makes logging a main performance issue. To overcome this problem, this is a add-on to jdk logging to put log entries on a concurrent message queue and log the messages one by one using a separate process. - FTPClient uses the concurrent logging instead of the log4j logger
2013-07-09 14:28:25 +02:00
ConcurrentLog.warn("heuristicRSS", "url not well-formed: '" + urlString + "'");
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
return;
}
// if we have an url then try to load the rss
RSSReader rss = null;
complete redesign of search process: - removed 'worker' processes - no internal time-out behaviour: methods either are successful or return null - waiting is only done on top-level - removed snippet-production; this is replaced by solr snippets - removed statistics based on solr size queries (they had been VERY long); the statistics (like suggestions or tag cloud) are now again based on the old but very fast RWI index. In portal or intranet mode the RWI index is usually switched off; if you like to have statistics again then you must switch on the rwis again in this mode. - fixed many bugs regarding correct page counter
2013-02-26 17:16:31 +01:00
searchEvent.oneFeederStarted();
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
try {
final Response response =
Redesign of crawler identification and robots steering. A non-p2p user in intranets and the internet can now choose to appear as Googlebot. This is an essential necessity to be able to compete in the field of commercial search appliances, since most web pages are these days optimized only for Google and no other search platform any more. All commercial search engine providers have a built-in fake-Google User Agent to be able to get the same search index as Google can do. Without the resistance against obeying to robots.txt in this case, no competition is possible any more. YaCy will always obey the robots.txt when it is used for crawling the web in a peer-to-peer network, but to establish a Search Appliance (like a Google Search Appliance, GSA) it is necessary to be able to behave exactly like a Google crawler. With this change, you will be able to switch the user agent when portal or intranet mode is selected on per-crawl-start basis. Every crawl start can have a different user agent.
2013-08-22 14:23:47 +02:00
Switchboard.this.loader.load(Switchboard.this.loader.request(url, true, false), CacheStrategy.NOCACHE, BlacklistType.SEARCH, ClientIdentification.yacyIntranetCrawlerAgent);
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
final byte[] resource = (response == null) ? null : response.getContent();
//System.out.println("BLEKKO: " + UTF8.String(resource));
rss = resource == null ? null : RSSReader.parse(RSSFeed.DEFAULT_MAXSIZE, resource);
if ( rss != null ) {
- the webgraph shall store all links which appear on a web page and not all unique links! This made it necessary, that a large portion of the parser and link processing classes must be adopted to carry a different type of link collection which carry a property attribute which are attached to web anchors. - introduction of a new URL class, AnchorURL - the other url classes, DigestURI and MultiProtocolURI had been renamed and refactored to fit into a new document package schema, document.id - cleanup of net.yacy.cora.document package and refactoring
2013-09-15 00:30:23 +02:00
final Map<DigestURL, String> links = new TreeMap<DigestURL, String>();
DigestURL uri;
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
for ( final RSSMessage message : rss.getFeed() ) {
try {
- the webgraph shall store all links which appear on a web page and not all unique links! This made it necessary, that a large portion of the parser and link processing classes must be adopted to carry a different type of link collection which carry a property attribute which are attached to web anchors. - introduction of a new URL class, AnchorURL - the other url classes, DigestURI and MultiProtocolURI had been renamed and refactored to fit into a new document package schema, document.id - cleanup of net.yacy.cora.document package and refactoring
2013-09-15 00:30:23 +02:00
uri = new DigestURL(message.getLink());
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
links.put(uri, message.getTitle());
Added 'final' for all exception blocks as this helps the Java compiler to optimize memory usage Conflicts: source/net/yacy/search/Switchboard.java
2013-07-17 18:31:30 +02:00
} catch (final MalformedURLException e ) {
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
}
}
- refactoring of log to ConcurrentLog: jdk-based logger tend to block at java.util.logging.Logger.log(Logger.java:476) in concurrent environments. This makes logging a main performance issue. To overcome this problem, this is a add-on to jdk logging to put log entries on a concurrent message queue and log the messages one by one using a separate process. - FTPClient uses the concurrent logging instead of the log4j logger
2013-07-09 14:28:25 +02:00
ConcurrentLog.info("heuristicRSS", "Heuristic: adding "
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
+ links.size()
+ " links from '"
+ feedName
+ "' rss feed");
// add all pages to the index
added a feature to set a collection for a crawl result based on a regular expression on th url: the collection attribut for a crawl start may be now either a token or a list of tokens, seperated by ',' where a token is either a string or a pair <string,pattern> where the string is separated to the pattern with a ':' and the string is assigned to the document as collection only if the pattern matches with the url.
2013-08-25 00:13:48 +02:00
addAllToIndex(null, links, searchEvent, feedName, CrawlProfile.collectionParser("rss"));
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
}
Added 'final' for all exception blocks as this helps the Java compiler to optimize memory usage Conflicts: source/net/yacy/search/Switchboard.java
2013-07-17 18:31:30 +02:00
} catch (final Throwable e ) {
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
//Log.logException(e);
} finally {
complete redesign of search process: - removed 'worker' processes - no internal time-out behaviour: methods either are successful or return null - waiting is only done on top-level - removed snippet-production; this is replaced by solr snippets - removed statistics based on solr size queries (they had been VERY long); the statistics (like suggestions or tag cloud) are now again based on the old but very fast RWI index. In portal or intranet mode the RWI index is usually switched off; if you like to have statistics again then you must switch on the rwis again in this mode. - fixed many bugs regarding correct page counter
2013-02-26 17:16:31 +01:00
searchEvent.oneFeederTerminated();
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
}
}
}.start();
}
public static int currentPPM() {
return EventTracker.countEvents(EventTracker.EClass.INDEX, 20000) * 3;
}
public float averageQPM() {
final long uptime = (System.currentTimeMillis() - serverCore.startupTime) / 1000;
return (this.searchQueriesRobinsonFromRemote + this.searchQueriesGlobal) * 60f / Math.max(uptime, 1f);
}
public float averageQPMGlobal() {
final long uptime = (System.currentTimeMillis() - serverCore.startupTime) / 1000;
return (this.searchQueriesGlobal) * 60f / Math.max(uptime, 1f);
}
public float averageQPMPrivateLocal() {
final long uptime = (System.currentTimeMillis() - serverCore.startupTime) / 1000;
return (this.searchQueriesRobinsonFromLocal) * 60f / Math.max(uptime, 1f);
}
public float averageQPMPublicLocal() {
final long uptime = (System.currentTimeMillis() - serverCore.startupTime) / 1000;
return (this.searchQueriesRobinsonFromRemote) * 60f / Math.max(uptime, 1f);
}
prevent that the size of the index is computed too many times. Because the index size is now provided by solr, and the only way to do that is a match for [* TO *], a size computation is quite complex and time-consuming. Therefore this patch prevents that the method is called at all and if necessary puts a DOS-preventing barrier in front of it.
2013-05-08 11:50:46 +02:00
private static long indeSizeCache = 0;
private static long indexSizeTime = 0;
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
public void updateMySeed() {
remove dependency from old serverCore.java - remaining getPortNr not needed (as current release allows only to set plain integer as port, see ConfigBasic)
2013-12-29 02:00:44 +01:00
this.peers.mySeed().put(Seed.PORT, getConfig("port", "8090"));
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
//the speed of indexing (pages/minute) of the peer
final long uptime = (System.currentTimeMillis() - serverCore.startupTime) / 1000;
- added ssl configuration sign (a lock) to network statistic/table - fixed a bug in bitfield
2013-05-10 17:32:21 +02:00
Seed mySeed = this.peers.mySeed();
mySeed.put(Seed.ISPEED, Integer.toString(currentPPM()));
mySeed.put(Seed.RSPEED, Float.toString(averageQPM()));
mySeed.put(Seed.UPTIME, Long.toString(uptime / 60)); // the number of minutes that the peer is up in minutes/day (moving average MA30)
prevent that the size of the index is computed too many times. Because the index size is now provided by solr, and the only way to do that is a match for [* TO *], a size computation is quite complex and time-consuming. Therefore this patch prevents that the method is called at all and if necessary puts a DOS-preventing barrier in front of it.
2013-05-08 11:50:46 +02:00
long t = System.currentTimeMillis();
if (t - indexSizeTime > 60000) {
indeSizeCache = sb.index.fulltext().collectionSize();
indexSizeTime = t;
}
- added ssl configuration sign (a lock) to network statistic/table - fixed a bug in bitfield
2013-05-10 17:32:21 +02:00
mySeed.put(Seed.LCOUNT, Long.toString(indeSizeCache)); // the number of links that the peer has stored (LURL's)
mySeed.put(Seed.NCOUNT, Integer.toString(this.crawlQueues.noticeURL.size())); // the number of links that the peer has noticed, but not loaded (NURL's)
mySeed.put(
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
Seed.RCOUNT,
Integer.toString(this.crawlQueues.noticeURL.stackSize(NoticedURL.StackType.GLOBAL))); // the number of links that the peer provides for remote crawling (ZURL's)
- added ssl configuration sign (a lock) to network statistic/table - fixed a bug in bitfield
2013-05-10 17:32:21 +02:00
mySeed.put(Seed.ICOUNT, Long.toString(this.index.RWICount())); // the minimum number of words that the peer has indexed (as it says)
mySeed.put(Seed.SCOUNT, Integer.toString(this.peers.sizeConnected())); // the number of seeds that the peer has stored
mySeed.put(
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
Seed.CCOUNT,
Float.toString(((int) ((this.peers.sizeConnected() + this.peers.sizeDisconnected() + this.peers
.sizePotential()) * 60.0f / (uptime + 1.01f)) * 100.0f) / 100.0f)); // the number of clients that the peer connects (as connects/hour)
- added ssl configuration sign (a lock) to network statistic/table - fixed a bug in bitfield
2013-05-10 17:32:21 +02:00
mySeed.put(Seed.VERSION, yacyBuildProperties.getLongVersion());
mySeed.setFlagDirectConnect(true);
mySeed.setLastSeenUTC();
mySeed.put(Seed.UTC, GenericFormatter.UTCDiffString());
mySeed.setFlagAcceptRemoteCrawl(getConfigBool("crawlResponse", true));
mySeed.setFlagAcceptRemoteIndex(getConfigBool("allowReceiveIndex", true));
fix for strange NPE java.lang.NullPointerException at net.yacy.search.Switchboard.updateMySeed(Switchboard.java:3667) at net.yacy.peers.Network.peerPing(Network.java:195) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:616) at net.yacy.kelondro.workflow.InstantBusyThread.job(InstantBusyThread.java:107) at net.yacy.kelondro.workflow.AbstractBusyThread.run(AbstractBusyThread.java:165)
2013-12-29 00:40:31 +01:00
mySeed.setFlagSSLAvailable(this.getHttpServer() != null && this.getHttpServer().withSSL() && getConfigBool("server.https", false));
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
}
public void loadSeedLists() {
// uses the superseed to initialize the database with known seeds
String seedListFileURL;
final int sc = this.peers.sizeConnected();
- refactoring of log to ConcurrentLog: jdk-based logger tend to block at java.util.logging.Logger.log(Logger.java:476) in concurrent environments. This makes logging a main performance issue. To overcome this problem, this is a add-on to jdk logging to put log entries on a concurrent message queue and log the messages one by one using a separate process. - FTPClient uses the concurrent logging instead of the log4j logger
2013-07-09 14:28:25 +02:00
Network.log.info("BOOTSTRAP: " + sc + " seeds known from previous run, concurrently starting seedlist loader");
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
// - use the superseed to further fill up the seedDB
AtomicInteger scc = new AtomicInteger(0);
int c = 0;
while ( true ) {
if ( Thread.currentThread().isInterrupted() ) {
break;
}
Redesign of crawler identification and robots steering. A non-p2p user in intranets and the internet can now choose to appear as Googlebot. This is an essential necessity to be able to compete in the field of commercial search appliances, since most web pages are these days optimized only for Google and no other search platform any more. All commercial search engine providers have a built-in fake-Google User Agent to be able to get the same search index as Google can do. Without the resistance against obeying to robots.txt in this case, no competition is possible any more. YaCy will always obey the robots.txt when it is used for crawling the web in a peer-to-peer network, but to establish a Search Appliance (like a Google Search Appliance, GSA) it is necessary to be able to behave exactly like a Google crawler. With this change, you will be able to switch the user agent when portal or intranet mode is selected on per-crawl-start basis. Every crawl start can have a different user agent.
2013-08-22 14:23:47 +02:00
seedListFileURL = this.getConfig(SwitchboardConstants.NETWORK_BOOTSTRAP_SEEDLIST_STUB + c, "");
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
if ( seedListFileURL.isEmpty() ) {
break;
}
c++;
if ( seedListFileURL.startsWith("http://") || seedListFileURL.startsWith("https://") ) {
When doing bootstraping, always accept one seedlist-File without checking the date of the file. This should help to start the peer in case that the user has a completely wrong date setting.
2013-10-22 15:34:51 +02:00
loadSeedListConcurrently(this.peers, seedListFileURL, scc, (int) getConfigLong("bootstrapLoadTimeout", 20000), c > 0);
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
}
}
}
When doing bootstraping, always accept one seedlist-File without checking the date of the file. This should help to start the peer in case that the user has a completely wrong date setting.
2013-10-22 15:34:51 +02:00
private static void loadSeedListConcurrently(final SeedDB peers, final String seedListFileURL, final AtomicInteger scc, final int timeout, final boolean checkAge) {
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
// uses the superseed to initialize the database with known seeds
Thread seedLoader = new Thread() {
@Override
public void run() {
// load the seed list
try {
- the webgraph shall store all links which appear on a web page and not all unique links! This made it necessary, that a large portion of the parser and link processing classes must be adopted to carry a different type of link collection which carry a property attribute which are attached to web anchors. - introduction of a new URL class, AnchorURL - the other url classes, DigestURI and MultiProtocolURI had been renamed and refactored to fit into a new document package schema, document.id - cleanup of net.yacy.cora.document package and refactoring
2013-09-15 00:30:23 +02:00
DigestURL url = new DigestURL(seedListFileURL);
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
//final long start = System.currentTimeMillis();
final RequestHeader reqHeader = new RequestHeader();
reqHeader.put(HeaderFramework.PRAGMA, "no-cache");
reqHeader.put(HeaderFramework.CACHE_CONTROL, "no-cache");
Redesign of crawler identification and robots steering. A non-p2p user in intranets and the internet can now choose to appear as Googlebot. This is an essential necessity to be able to compete in the field of commercial search appliances, since most web pages are these days optimized only for Google and no other search platform any more. All commercial search engine providers have a built-in fake-Google User Agent to be able to get the same search index as Google can do. Without the resistance against obeying to robots.txt in this case, no competition is possible any more. YaCy will always obey the robots.txt when it is used for crawling the web in a peer-to-peer network, but to establish a Search Appliance (like a Google Search Appliance, GSA) it is necessary to be able to behave exactly like a Google crawler. With this change, you will be able to switch the user agent when portal or intranet mode is selected on per-crawl-start basis. Every crawl start can have a different user agent.
2013-08-22 14:23:47 +02:00
final HTTPClient client = new HTTPClient(ClientIdentification.yacyInternetCrawlerAgent, timeout);
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
client.setHeader(reqHeader.entrySet());
client.HEADResponse(url.toString());
int statusCode = client.getHttpResponse().getStatusLine().getStatusCode();
ResponseHeader header = new ResponseHeader(statusCode, client.getHttpResponse().getAllHeaders());
When doing bootstraping, always accept one seedlist-File without checking the date of the file. This should help to start the peer in case that the user has a completely wrong date setting.
2013-10-22 15:34:51 +02:00
if (checkAge) {
if ( header.lastModified() == null ) {
Network.log.warn("BOOTSTRAP: seed-list URL "
+ seedListFileURL
+ " not usable, last-modified is missing");
return;
} else if ( (header.age() > 86400000) && (scc.get() > 0) ) {
Network.log.info("BOOTSTRAP: seed-list URL "
+ seedListFileURL
+ " too old ("
+ (header.age() / 86400000)
+ " days)");
return;
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
}
When doing bootstraping, always accept one seedlist-File without checking the date of the file. This should help to start the peer in case that the user has a completely wrong date setting.
2013-10-22 15:34:51 +02:00
}
scc.incrementAndGet();
fixed shutdown authenticaton problem
2014-01-06 01:48:54 +01:00
final byte[] content = client.GETbytes(url, null);
When doing bootstraping, always accept one seedlist-File without checking the date of the file. This should help to start the peer in case that the user has a completely wrong date setting.
2013-10-22 15:34:51 +02:00
Iterator<String> enu = FileUtils.strings(content);
int lc = 0;
while ( enu.hasNext() ) {
try {
Seed ys = Seed.genRemoteSeed(enu.next(), false, null);
if ( (ys != null)
&& (!peers.mySeedIsDefined() || !peers.mySeed().hash.equals(ys.hash)) ) {
final long lastseen = Math.abs((System.currentTimeMillis() - ys.getLastSeenUTC()) / 1000 / 60);
if ( lastseen < 60 ) {
if ( peers.peerActions.connectPeer(ys, false) ) {
lc++;
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
}
}
}
When doing bootstraping, always accept one seedlist-File without checking the date of the file. This should help to start the peer in case that the user has a completely wrong date setting.
2013-10-22 15:34:51 +02:00
} catch (final IOException e ) {
Network.log.info("BOOTSTRAP: bad seed from " + seedListFileURL + ": " + e.getMessage());
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
}
}
When doing bootstraping, always accept one seedlist-File without checking the date of the file. This should help to start the peer in case that the user has a completely wrong date setting.
2013-10-22 15:34:51 +02:00
Network.log.info("BOOTSTRAP: "
+ lc
+ " seeds from seed-list URL "
+ seedListFileURL
+ ", AGE="
+ (header.age() / 3600000)
+ "h");
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
Added 'final' for all exception blocks as this helps the Java compiler to optimize memory usage Conflicts: source/net/yacy/search/Switchboard.java
2013-07-17 18:31:30 +02:00
} catch (final IOException e ) {
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
// this is when wget fails, commonly because of timeout
- refactoring of log to ConcurrentLog: jdk-based logger tend to block at java.util.logging.Logger.log(Logger.java:476) in concurrent environments. This makes logging a main performance issue. To overcome this problem, this is a add-on to jdk logging to put log entries on a concurrent message queue and log the messages one by one using a separate process. - FTPClient uses the concurrent logging instead of the log4j logger
2013-07-09 14:28:25 +02:00
Network.log.info("BOOTSTRAP: failed (1) to load seeds from seed-list URL "
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
+ seedListFileURL + ": " + e.getMessage());
Added 'final' for all exception blocks as this helps the Java compiler to optimize memory usage Conflicts: source/net/yacy/search/Switchboard.java
2013-07-17 18:31:30 +02:00
} catch (final Exception e ) {
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
// this is when wget fails; may be because of missing internet connection
- refactoring of log to ConcurrentLog: jdk-based logger tend to block at java.util.logging.Logger.log(Logger.java:476) in concurrent environments. This makes logging a main performance issue. To overcome this problem, this is a add-on to jdk logging to put log entries on a concurrent message queue and log the messages one by one using a separate process. - FTPClient uses the concurrent logging instead of the log4j logger
2013-07-09 14:28:25 +02:00
Network.log.severe("BOOTSTRAP: failed (2) to load seeds from seed-list URL "
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
+ seedListFileURL + ": " + e.getMessage(), e);
}
}
};
seedLoader.start();
}
public void initRemoteProxy() {
// reading the proxy host name
final String host = getConfig("remoteProxyHost", "").trim();
// reading the proxy host port
int port;
try {
port = Integer.parseInt(getConfig("remoteProxyPort", "3128"));
Added 'final' for all exception blocks as this helps the Java compiler to optimize memory usage Conflicts: source/net/yacy/search/Switchboard.java
2013-07-17 18:31:30 +02:00
} catch (final NumberFormatException e ) {
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
port = 3128;
}
// create new config
ProxySettings.port = port;
ProxySettings.host = host;
ProxySettings.setProxyUse4HTTP(ProxySettings.host != null && ProxySettings.host.length() > 0 && getConfigBool("remoteProxyUse", false));
ProxySettings.setProxyUse4YaCy(getConfig("remoteProxyUse4Yacy", "true").equalsIgnoreCase("true"));
ProxySettings.setProxyUse4HTTPS(getConfig("remoteProxyUse4SSL", "true").equalsIgnoreCase("true"));
ProxySettings.user = getConfig("remoteProxyUser", "").trim();
ProxySettings.password = getConfig("remoteProxyPwd", "").trim();
// determining addresses for which the remote proxy should not be used
final String remoteProxyNoProxy = getConfig("remoteProxyNoProxy", "").trim();
ProxySettings.noProxy = remoteProxyNoProxy.split(",");
// trim split entries
int i = 0;
for ( final String pattern : ProxySettings.noProxy ) {
ProxySettings.noProxy[i] = pattern.trim();
i++;
}
}
public void checkInterruption() throws InterruptedException {
final Thread curThread = Thread.currentThread();
if ( (curThread instanceof WorkflowThread) && ((WorkflowThread) curThread).shutdownInProgress() ) {
throw new InterruptedException("Shutdown in progress ...");
} else if ( this.terminate || curThread.isInterrupted() ) {
throw new InterruptedException("Shutdown in progress ...");
}
}
public void terminate(final long delay, final String reason) {
if ( delay <= 0 ) {
throw new IllegalArgumentException("The shutdown delay must be greater than 0.");
}
- refactoring of log to ConcurrentLog: jdk-based logger tend to block at java.util.logging.Logger.log(Logger.java:476) in concurrent environments. This makes logging a main performance issue. To overcome this problem, this is a add-on to jdk logging to put log entries on a concurrent message queue and log the messages one by one using a separate process. - FTPClient uses the concurrent logging instead of the log4j logger
2013-07-09 14:28:25 +02:00
this.log.info("caught delayed terminate request: " + reason);
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
(new Shutdown(this, delay, reason)).start();
}
public boolean shallTerminate() {
return this.terminate;
}
public void terminate(final String reason) {
this.terminate = true;
- refactoring of log to ConcurrentLog: jdk-based logger tend to block at java.util.logging.Logger.log(Logger.java:476) in concurrent environments. This makes logging a main performance issue. To overcome this problem, this is a add-on to jdk logging to put log entries on a concurrent message queue and log the messages one by one using a separate process. - FTPClient uses the concurrent logging instead of the log4j logger
2013-07-09 14:28:25 +02:00
this.log.info("caught terminate request: " + reason);
fix: exception if default work files don't exist
2012-12-22 23:03:39 +01:00
this.shutdownSync.release();
}
public boolean isTerminated() {
return this.terminate;
}
public boolean waitForShutdown() throws InterruptedException {
this.shutdownSync.acquire();
return this.terminate;
}
}
Reference in New Issue Copy Permalink