Crawler crawlt nicht mehr

Hier finden YaCy User Hilfe wenn was nicht funktioniert oder anders funktioniert als man dachte. Bei offensichtlichen Fehlern diese bitte gleich in die Bugs (http://bugs.yacy.net) eintragen.
Forumsregeln
In diesem Forum geht es um Benutzungsprobleme und Anfragen für Hilfe. Wird dabei ein Bug identifiziert, wird der thread zur Bearbeitung in die Bug-Sektion verschoben. Wer hier also einen Thread eingestellt hat und ihn vermisst, wird ihn sicherlich in der Bug-Sektion wiederfinden.

Crawler crawlt nicht mehr

Beitragvon Icebreeze » Fr Mär 13, 2009 6:01 pm

Hallo zusammen

Mein Crawler crawlt nicht mehr. Selbst neu gestartete Crawls mit autoReCrawl/monthly/ oder autoReCrawl/Weekly werden einmal abgearbeitet und dann terminiert. Das sieht dann so aus:

Bild

Hatte bis zu dem Zeitpunkt als dies aufgetreten ist an die 100 Crawls (autoReCrawl/monthly/ oder autoReCrawl/Weekly) am Laufen. Plötzlich waren alle terminiert. OK, alle gelöscht und die im Bild neu gestartet. Nach dem Abarbeiten war wieder Sense. Woran kann das liegen?

rgds
IceBreeze
Icebreeze
 
Beiträge: 30
Registriert: Do Okt 16, 2008 6:37 pm

Re: Crawler crawlt nicht mehr

Beitragvon Icebreeze » So Mär 15, 2009 8:07 am

Hi

Hm, ok, ein paar Infos mehr wären sicher nicht schlecht.
In der zwischenzeit habe ich mehrere Crawls gestartet (immer nach dem Schema Lesezeichen erstellen->autoReCrawl/monthly/ oder autoReCrawl/Weekly). Die werde alle gecrawlt, aber sobald sie fertig sind, werden sie terminiert. Früher waren die imho immer aktiv, auch wenn sie nicht gerade crawlten.

rgds
IceBreeze

Hier die Erweiterten Einstellungen:

10_httpd_busysleep: 0
10_httpd_idlesleep: 0
10_httpd_memprereq: 0
20_dhtdistribution_busysleep: 2000
20_dhtdistribution_idlesleep: 30000
20_dhtdistribution_memprereq: 6291456
30_peerping_busysleep: 120000
30_peerping_idlesleep: 120000
30_peerping_memprereq: 1048576
40_peerseedcycle_busysleep: 1200000
40_peerseedcycle_idlesleep: 1800000
40_peerseedcycle_memprereq: 2097152
50_localcrawl_busysleep: 10
50_localcrawl_idlesleep: 2000
50_localcrawl_isPaused: false
50_localcrawl_memprereq: 4194304
60_remotecrawlloader_busysleep: 5000
60_remotecrawlloader_idlesleep: 10000
60_remotecrawlloader_isPaused: false
60_remotecrawlloader_memprereq: 2097152
62_remotetriggeredcrawl_busysleep: 1000
62_remotetriggeredcrawl_idlesleep: 3000
62_remotetriggeredcrawl_isPaused: false
62_remotetriggeredcrawl_memprereq: 6291456
80_indexing_busysleep: 1
80_indexing_idlesleep: 1000
80_indexing_memprereq: 6291456
85_cacheflush_busysleep: 10000
85_cacheflush_idlesleep: 60000
85_cacheflush_memprereq: 0
90_cleanup_busysleep: 300000
90_cleanup_idlesleep: 300000
90_cleanup_memprereq: 0
BlackLists.DefaultList: url.default.black
BlackLists.Shared: url.default.black
BlackLists.class: de.anomic.index.indexDefaultReferenceBlacklist
CRDist0Method: 1
CRDist0Path: GLOBAL/010_owncr
CRDist0Percent: 0
CRDist0Target:
CRDist1Method: 9
CRDist1Path: GLOBAL/014_othercr
CRDist1Percent: 30
CRDist1Target: kaskelix.de:8080,yacy.dyndns.org:8000
CRDistOn: true
WikiAccess: admin
YaCyHop: true
adminAccount:
adminAccountBase64MD5: ein Hash
adminAccountForLocalhost: false
allowDistributeIndex: true
allowDistributeIndexWhileCrawling: false
allowDistributeIndexWhileIndexing: true
allowReceiveIndex: false
allowUnlimitedReceiveIndexFrom:
applicationRoot: /usr/share/yacy
autoReCrawl_busysleep: 3600000
autoReCrawl_idlesleep: 3600000
autoReCrawl_memprereq: -1
bindPort:
bootstrapLoadTimeout: 6000
browserPopUpApplication: firefox
browserPopUpPage: yacyinteractive.html?display=2
browserPopUpTrigger: true
cgi.allow: false
cgi.suffixes: cgi,pl
cleanup.deletionProcessedNews: true
cleanup.deletionPublishedNews: true
clientTimeout: 10000
cluster.mode: publicpeer
cluster.peers.ipport:
cluster.peers.yacydomain: localpeer.yacy
compare_yacy.left: YaCy
compare_yacy.right: google.de
connectionKeepAliveSupport: true
crawlOrder: false
crawlOrderDelay: 8
crawlOrderDepth: 4
crawlPause.localsearch: 9000
crawlPause.proxy: 15000
crawlPause.remotesearch: 3000
crawlResponse: false
crawlResponseDepth: 0
crawler.BlackLists: url.default.black
crawler.MaxActiveThreads: 30
crawler.clientTimeout: 9000
crawler.ftp.maxFileSize: 262144
crawler.http.acceptCharset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
crawler.http.acceptEncoding: gzip
crawler.http.acceptLanguage: en-us,en;q=0.5
crawler.http.maxFileSize: 262144
crawlingDepth: 3
crawlingDomFilterDepth: -1
crawlingDomMaxPages: -1
crawlingFilter: .*
crawlingIfOlder: 1229279757359
crawlingQ: true
currentSkin: default
dbPath: DATA/PLASMADB
defaultFiles: index.html,index.htm,default.html,search.html,console.html,control.html,welcome.html,wiki.html,forum.html,blog.html,email.html,content.html,monitor.html,share.html,dir.html,readme.txt
defaultLinkReceiveFrequency: 30
defaultWordReceiveFrequency: 100
dht.BlackLists: url.default.black
disk.free: 3000
enableTemplateCache: true
externalRedirector:
fileHost: localpeer
filterOutStopwordsFromTopwords: true
htDefaultPath: htroot
htDocsPath: DATA/HTDOCS
htRootPath: htroot
htTemplatePath: htroot/env/templates
httpc.nameCacheNoCachingPatterns: .*.ath.cx,.*.blogdns.*,.*.boldlygoingnowhere.org,.*.dnsalias.*,.*.dnsdojo.*,.*.dvrdns.org,.*.dyn-o-saur.com,.*.dynalias.*,.*.dyndns.*,.*.ftpaccess.cc,.*.game-host.org,.*.game-server.cc,.*.getmyip.com,.*.gotdns.*,.*.ham-radio-op.net,.*.hobby-site.com,.*.homedns.org,.*.homeftp.*,.*.homeip.net,.*.homelinux.*,.*.homeunix.*,.*.is-a-chef.*,.*.is-a-geek.*,.*.kicks-ass.*,.*.merseine.nu,.*.mine.nu,.*.myphotos.cc,.*.podzone.*,.*.scrapping.cc,.*.selfip.*,.*.servebbs.*,.*.serveftp.*,.*.servegame.org,.*.shacknet.nu
httpd.robots.txt: locked,dirs
httpdMaxBusySessions: 100
index.storeCommons: false
indexControl.gzipBody: true
indexControl.timeout: 60000
indexDistribution.gzipBody: true
indexDistribution.maxChunkFails: 1
indexDistribution.maxChunkSize: 1000
indexDistribution.maxOpenFiles: 800
indexDistribution.minChunkSize: 10
indexDistribution.startChunkSize: 200
indexDistribution.timeout: 60000
indexMedia: true
indexPrimaryPath: DATA/INDEX
indexReceiveBlockBlacklist: true
indexSecondaryPath:
indexText: true
indexTransfer.gzipBody: true
indexTransfer.maxOpenFiles: 800
indexTransfer.timeout: 120000
indexer.slots: 40
isTransparentProxy: false
javastart_Xms: Xms640m
javastart_Xmx: Xmx640m
javastart_priority: 0
keyStore:
keyStorePassword:
listsPath: DATA/LISTS
locale.lang: default/English,de/Deutsch,fr/Français,nl/Nederlands,it/Italiano,es/Español,pt/Portugês,fi/Suomi,se/Svenska,dk/Dansk,gr/Eλληvικα,sk/Slovensky
locale.language: de
locale.source: locales
locale.translated_html: DATA/LOCALE/htroot
locale.work: DATA/LOCALE/locales
mediaExt: 7z,ace,aif,aiff,arj,asf,asx,avi,bin,bmp,bz2,css,db,dcm,deb,doc,dll,dmg,exe,gif,gz,hqx,ico,img,iso,jar,jpe,jpg,jpeg,lx,lxl,m4v,mpeg,mov,mp3,mpg,ogg,png,pdf,ppt,ps,ram,rar,rm,rpm,scr,sit,so,swf,sxc,sxd,sxi,sxw,tar,tbz,tgz,torrent,war,wav,wmv,xcf,xls,zip
memoryFreeAfterInitAGC: 545728056
memoryFreeAfterInitBGC: 540886568
memoryFreeAfterStartup: 634667544
memoryTotalAfterInitAGC: 667746304
memoryTotalAfterInitBGC: 667746304
memoryTotalAfterStartup: 635879424
mimeConfig: httpd.mime
minimumGlobalDelta: 500
minimumLocalDelta: 0
msgForwardingCmd: /usr/sbin/sendmail
msgForwardingEnabled: false
msgForwardingTo: root@localhost
network.group.definition: defaults/yacy.network.group
network.unit.access.blacklist:
network.unit.access.whitelist: 10\..*,127.*,172.(1[6-9]|2[0-9]|3[0-1])\..*,169.254.*,192.168.*,localhost
network.unit.bootstrap.seedlist0: http://www.yacy.net/seed.txt
network.unit.bootstrap.seedlist1: http://home.arcor.de/hermens/yacy/seed.txt
network.unit.bootstrap.seedlist2: http://low.audioattack.de/yacy/seed.txt
network.unit.bootstrap.seedlist3: http://www.lulabad.de/seed.txt
network.unit.definition: defaults/yacy.network.freeworld.unit
network.unit.description: Public YaCy Community
network.unit.dht: true
network.unit.dht.partitionExponent: 4
network.unit.dhtredundancy.junior: 1
network.unit.dhtredundancy.senior: 3
network.unit.domain: global
network.unit.name: freeworld
network.unit.protocol.control: uncontrolled
network.unit.remotecrawl.speed: 6
network.unit.search.time: 4
network.unit.update.location0: http://yacy.net/index.html
network.unit.update.location1: http://latest.yacy.de
network.unit.update.location2: http://www.findenstattsuchen.info/YaCy/latest/index.php
network.unit.update.location3: http://www.yacystats.de/yacybuild/
news.BlackLists: url.default.black
parseableExt: html,htm,txt,php,shtml,asp
parseableMimeTypes:
parseableMimeTypes.CRAWLER: application/atom+xml,application/bzip2,application/excel,application/gzip,application/java-archive,application/msexcel,application/mspowerpoint,application/msword,application/postscript,application/powerpoint,application/rdf+xml,application/rss+xml,application/rtf,application/vcard,application/vnd.ms-excel,application/vnd.ms-powerpoint,application/x-7z-compressed,application/x-bz2,application/x-bzip2,application/x-excel,application/x-gzip,application/x-msexcel,application/x-zip,application/x-zip-compressed,application/xml,application/zip,text/postscript,text/rss,text/rtf,text/x-vcard,text/xml
parseableMimeTypes.HTML: application/xhtml+xml,text/html,text/plain,text/sgml
parseableMimeTypes.ICAP: application/atom+xml,application/bzip2,application/excel,application/gzip,application/java-archive,application/msexcel,application/mspowerpoint,application/msword,application/postscript,application/powerpoint,application/rdf+xml,application/rss+xml,application/rtf,application/vcard,application/vnd.ms-excel,application/vnd.ms-powerpoint,application/x-7z-compressed,application/x-bz2,application/x-bzip2,application/x-excel,application/x-gzip,application/x-msexcel,application/x-zip,application/x-zip-compressed,application/xml,application/zip,text/postscript,text/rss,text/rtf,text/x-vcard,text/xml
parseableMimeTypes.IMAGE: application/atom+xml,application/bzip2,application/excel,application/gzip,application/java-archive,application/msexcel,application/mspowerpoint,application/msword,application/postscript,application/powerpoint,application/rdf+xml,application/rss+xml,application/rtf,application/vcard,application/vnd.ms-excel,application/vnd.ms-powerpoint,application/x-7z-compressed,application/x-bz2,application/x-bzip2,application/x-excel,application/x-gzip,application/x-msexcel,application/x-zip,application/x-zip-compressed,application/xml,application/zip,text/postscript,text/rss,text/rtf,text/x-vcard,text/xml
parseableMimeTypes.PROXY: application/atom+xml,application/bzip2,application/excel,application/gzip,application/java-archive,application/msexcel,application/mspowerpoint,application/msword,application/postscript,application/powerpoint,application/rdf+xml,application/rss+xml,application/rtf,application/vcard,application/vnd.ms-excel,application/vnd.ms-powerpoint,application/x-7z-compressed,application/x-bz2,application/x-bzip2,application/x-excel,application/x-gzip,application/x-msexcel,application/x-zip,application/x-zip-compressed,application/xml,application/zip,text/postscript,text/rss,text/rtf,text/x-vcard,text/xml
parseableMimeTypes.URLREDIRECTOR: application/atom+xml,application/bzip2,application/excel,application/gzip,application/java-archive,application/msexcel,application/mspowerpoint,application/msword,application/postscript,application/powerpoint,application/rdf+xml,application/rss+xml,application/rtf,application/vcard,application/vnd.ms-excel,application/vnd.ms-powerpoint,application/x-7z-compressed,application/x-bz2,application/x-bzip2,application/x-excel,application/x-gzip,application/x-msexcel,application/x-zip,application/x-zip-compressed,application/xml,application/zip,text/postscript,text/rss,text/rtf,text/x-vcard,text/xml
peerCycle: 2
peerName: aquayacy
performanceIO: 10
performanceProfile: defaults/yacy.init
performanceSpeed: 100
pkcs12ImportFile:
pkcs12ImportPwd:
plasmaBlueList: yacy.blue
port: 8081
promoteSearchPageGreeting: AquaYaCy
promoteSearchPageGreeting.homepage: http://yacy.net
promoteSearchPageGreeting.largeImage: /env/grafics/YaCyLogo_120ppi.png
promoteSearchPageGreeting.smallImage: /env/grafics/YaCyLogo_60ppi.png
promoteSearchPageGreeting.useNetworkName: false
proxy.BlackLists: url.default.black
proxy.clientTimeout: 30000
proxy.monitorCookies: false
proxy.sendViaHeader: true
proxy.sendXForwardedForHeader: true
proxyBlueList: yacy.blue
proxyCache: DATA/HTCACHE
proxyCacheSize: 100
proxyClient: localhost,127.0.0.1,192.168.*,10.*
proxyCookieBlackList: cookie.default.black
proxyCookieWhiteList: cookie.default.black
proxyIndexingLocalMedia: true
proxyIndexingLocalText: true
proxyIndexingRemote: false
proxyPrefetchDepth: 0
proxyStoreHTCache: true
proxyYellowList: yacy.yellow
publicSearchpage: true
publicSurftips: false
rankingPath: DATA/RANKING
rankingProfile:
releases: DATA/RELEASE
remoteProxyHost: 192.168.2.2
remoteProxyNoProxy: 192.*,10.*,127.*,localhost
remoteProxyPort: 4239
remoteProxyPwd:
remoteProxyUse: false
remoteProxyUse4SSL: true
remoteProxyUse4Yacy: true
remoteProxyUser:
repositoryPath: DATA/HTDOCS/repository
restart.cycle: 20
restart.hour: 03
restart.process: off
restart.time: 0
routing.deleteOldSeeds.permission: true
routing.deleteOldSeeds.time: 7
search.BlackLists: url.default.black
searchProcessLocalCount_c: 10000000
searchProcessLocalCount_f: 100
searchProcessLocalCount_j: 1000000
searchProcessLocalCount_o: 100
searchProcessLocalCount_r: 100000
searchProcessLocalCount_s: 30
searchProcessLocalCount_u: 10000
searchProcessLocalTime_c: 44
searchProcessLocalTime_f: 5
searchProcessLocalTime_j: 8
searchProcessLocalTime_o: 10
searchProcessLocalTime_r: 8
searchProcessLocalTime_s: 5
searchProcessLocalTime_u: 20
searchProcessRemoteCount_c: 1000000
searchProcessRemoteCount_f: 100
searchProcessRemoteCount_j: 1000000
searchProcessRemoteCount_o: 1000
searchProcessRemoteCount_r: 1000
searchProcessRemoteCount_s: 10
searchProcessRemoteCount_u: 1000
searchProcessRemoteTime_c: 44
searchProcessRemoteTime_f: 5
searchProcessRemoteTime_j: 8
searchProcessRemoteTime_o: 10
searchProcessRemoteTime_r: 8
searchProcessRemoteTime_s: 5
searchProcessRemoteTime_u: 20
secureHttps: true
seedFTPAccount:
seedFTPPassword:
seedFTPPath:
seedFTPServer:
seedFilePath:
seedScpAccount:
seedScpPassword:
seedScpPath:
seedScpServer:
seedScpServerPort:
seedUploadMethod: none
server.maxTrackingCount: 1000
server.maxTrackingHostCount: 100
server.maxTrackingTime: 3600000
serverAccount:
serverAccountBase64MD5:
serverClient: *
skinPath: DATA/SKINS
stacker.slots: 2000
staticIP:
storeHTCache: false
storeTXCache: true
surftips.BlackLists: url.default.black
svnRevision: 5632
thumbnailProgram:
timeout_media: 15000
timeout_text: 10000
trayIcon: true
update.blacklist: ...[123]
update.concept: any
update.cycle: 168
update.deleteOld: 30
update.process: manual
update.time.deploy: 0
update.time.download: 1236979774769
update.time.lookup: 1236979798570
upnp.enabled: false
useYacyReferer: true
use_proxyAccounts: false
vString: 0.720/05632
vdate: 20090221
version: 0.72005632
wikiParser.class: de.anomic.data.wikiCode
wordCacheInitCount: 30000
wordCacheMaxCount: 30000
workPath: DATA/WORK
xdstopw: false
xpstopw: false
xsstopw: true
yacyDebugMode: false
yacyStatus:
Icebreeze
 
Beiträge: 30
Registriert: Do Okt 16, 2008 6:37 pm


Zurück zu Fragen und Antworten

Wer ist online?

Mitglieder in diesem Forum: 0 Mitglieder und 2 Gäste

cron