Status autoupdate

Ereignisse, Vorschläge und Aktionen

Status autoupdate

Beitragvon datengrab » Mo Jun 09, 2008 10:33 am

Ich bin dabei Status.html zu überarbeiten, so dass alle Informationen wieder automatisch aktualisiert werden. Mein Ansatz ist es bisher alle relevanten Werte in <span></span> Blöcke zu fassen und diese dann per ajax zu aktuallisieren. Das funktioniert soweit auch ganz gut. Ist die Richtung richtig, oder habe da was übersehen?
datengrab
 
Beiträge: 366
Registriert: Sa Aug 25, 2007 7:22 pm

Re: Status autoupdate

Beitragvon Orbiter » Mo Jun 09, 2008 12:57 pm

Ist ja prima das du das machen möchtest! ich denke Ajax ist richtig.

Da Updaten ging ja füher mit einem Reload der ganzen Seite, das hatte ich aber raus gemacht weil damit das rssTerminal auch neu initialisiert werden würde, und dann funktioniert es nicht richtig. Mit Ajax verprellen wir zwar die 'no script' - Leute, aber da es sich hierbei nicht um eine Core-Funktion wie die Abfrage von Suchergebnissen handelt (wo es immer noch bzw. wieder ohne script geht), sondern 'nur' um Statusinfos, und diese darüberhinaus auch automatisch aktualisiert werden sollen kann man wohl nichts gegen Ajax sagen.

Bitte gucke mal in die htroot/xml-Dateien, da sind so viele Infos drin dass du wahrscheinlich keine eigenen Servelts schreiben musst um an die notwendigen Informationen für Ajax zu kommen.
Orbiter
 
Beiträge: 5796
Registriert: Di Jun 26, 2007 10:58 pm
Wohnort: Frankfurt am Main

Re: Status autoupdate

Beitragvon datengrab » Mo Jun 09, 2008 1:21 pm

Ja, in der status_p.xml steht schon einiges drin. Allerdings nicht alles, was ich brauche. Folglich habe ich sie ein wenig erweitert was denke ich besser ist als alles neu zu schreiben. Für die anzeige der Queues würde ich hingegen was neues schreiben, da das für status_p.xml zu viel wird und auf der Anderen Seite queues_p.xml overkill wäre. Oder ist es vom Performazegesichtspunkt besser doch eine eigene xml zu kreieren in der man dann alles auf einmal hat?
datengrab
 
Beiträge: 366
Registriert: Sa Aug 25, 2007 7:22 pm

Re: Status autoupdate

Beitragvon Orbiter » Mo Jun 09, 2008 2:30 pm

für die queues müsste es ebenfalls schon ein servlet in xml geben, weil das in WatchCrawler benutzt wird. Bitte da mal gucken.
Orbiter
 
Beiträge: 5796
Registriert: Di Jun 26, 2007 10:58 pm
Wohnort: Frankfurt am Main

Re: Status autoupdate

Beitragvon datengrab » Mo Jun 09, 2008 2:59 pm

Das ist das queues_p.xml. Aber in dem werden die kompletten Queues in die xml geschoben was natürlich ein riesen Aufwand ist. Für den Status benötige ich hingegen nur die Größe der Queues. Deswegen ist eigentlich die Frage, ob es sinnvoller ist die status_p.xml - die an sonsten nur von WatchCrawler mitverwendet wird - aufzublähen, eine "queues2_p.xml" nur für die Queuegrößen zu erstellen oder besser etwas nur für die Statusseite angepasstes zu bauen, wo sich dann alles in einer Datei findet.
datengrab
 
Beiträge: 366
Registriert: Sa Aug 25, 2007 7:22 pm

Re: Status autoupdate

Beitragvon Orbiter » Mo Jun 09, 2008 3:18 pm

ja dann würde ich sagen alles in die status_p.xml was du brauchst, kein extra Servlet machen. Sonst gibt das zu viele http-requests.
Orbiter
 
Beiträge: 5796
Registriert: Di Jun 26, 2007 10:58 pm
Wohnort: Frankfurt am Main

Re: Status autoupdate

Beitragvon datengrab » Do Jun 26, 2008 2:57 pm

Ich habe es jetzt so weit, das zu mindest die wichtigen Sachen auf Status.html automatisch aktuallisiert werden. Die Änderungen sind doch etwas umfangreicher, so das ich sie hier kurz vorstellen möchte, bevor ich sie commite.

Die Statusaktuallisierung und WatchCrawler benötigen teilweise die selben Daten. Da wir unnötige http-requests vermeiden wollen habe ich konsequenterweise beides so umgebaut, das beide nur noch eine xml anfordern. Die Dateien die die Daten bereitstellen sind fur Status:
xml/status_p.xml
xml/status_p.java
js/Status.js

und für WatchCrawler
xml/crawler_p.xml
xml/crawler_p.java
js/WatchCrawler.js

xml/queues_p.xml und xml/queues_p.java wurden um die Historie beizubehalten zu xml/crawler_p.xml und xml/crawler_p.java verschoben.

Gleichzeitig habe ich einige nicht gebrauchte Felder die für die xmls immer noch generiert wurden entfernt.

Und das sind alle Änderungen:
Code: Alles auswählen
Index: htroot/js/WatchCrawler.js
===================================================================
--- htroot/js/WatchCrawler.js   (Revision 4957)
+++ htroot/js/WatchCrawler.js   (Arbeitskopie)
@@ -3,69 +3,57 @@
BAR_IMG2="/env/grafics/red-block.png";
WORDCACHEBAR_LENGTH=1/4;

-
-var statusRPC;
-var queuesRPC;
+var crawlerRPC;
var refreshInterval=5;
var wait=0;
var changing=false; //change the interval

-refresh();
-//loadInterval=window.setInterval("refresh()", refreshInterval*1000);
+requestCrawler();
countInterval=window.setInterval("countdown()", 1000);

function changeInterval(){
   if(!changing){
      window.clearInterval(countInterval);
      counter=document.getElementById("nextUpdate");
-      //counter.innerHTML='<input type="text" id="newInterval" onblur="newInterval()" size="2" />';
-      //document.getElementById("newInterval").focus();
      counter.value=refreshInterval;
      changing=true;
   }
}
+
function newInterval(){
   refreshInterval=document.getElementById("nextUpdate").value;
-   refresh();
+   requestCrawler();
   countInterval=window.setInterval("countdown()", 1000);
   changing=false;
}
+
function countdown(){
   document.getElementById("nextUpdate").value=wait;
   wait--;
   if(wait==0){
-      refresh();
+      requestCrawler();
   }
}
-function refresh(){
-   wait=refreshInterval;
-   requestStatus();
-   requestQueues();
-}

-function requestStatus(){
-   statusRPC=createRequestObject();
-   statusRPC.open('get', '/xml/status_p.xml?html=');
-   statusRPC.onreadystatechange = handleStatus;
-   statusRPC.send(null);
+function requestCrawler(){
+    wait=refreshInterval;
+   crawlerRPC=createRequestObject();
+   crawlerRPC.open('get', '/xml/crawler_p.xml?html=');
+   crawlerRPC.onreadystatechange = handleCrawler;
+   crawlerRPC.send(null);
}
-function requestQueues(){
-   queuesRPC=createRequestObject();
-   queuesRPC.open('get', '/xml/queues_p.xml?html=');
-   queuesRPC.onreadystatechange = handleQueues;
-   queuesRPC.send(null);

-}
-
-function handleStatus(){
-    if(statusRPC.readyState != 4){
+function handleCrawler(){
+    if(crawlerRPC.readyState != 4){
      return;
   }
-   var statusResponse = statusRPC.responseXML;
-   statusTag=getFirstChild(statusResponse, "status");
+   var crawlerResponse = crawlerRPC.responseXML;
   
-   ppm=getValue(getFirstChild(statusTag, "ppm"));
+   crawlerTag = getFirstChild(crawlerResponse, "crawler");
+   statusTag = getFirstChild(crawlerTag, "status");
   
+   ppm = getValue(getFirstChild(statusTag, "ppm"));
+   
   var ppmNum = document.getElementById("ppmNum");
   removeAllChildren(ppmNum);
   ppmNum.appendChild(document.createTextNode(ppm));
@@ -94,10 +82,10 @@
   removeAllChildren(wordCacheNum);
   wordCacheNum.appendChild(document.createTextNode(wordCacheSize+"/"+wordCacheMaxSize));
   
-   wordCacheSpan=document.getElementById("wordcacheSpan");
+   wordCacheSpan = document.getElementById("wordcacheSpan");
   removeAllChildren(wordCacheSpan);
   var img;
-   var percent=Math.round(wordCache/wordCacheMax*100);
+   var percent = Math.round(wordCache/wordCacheMax*100);
   for(i=0;i<percent*WORDCACHEBAR_LENGTH;i++){
      img=document.createElement("img");
      img.setAttribute("src", BAR_IMG2);
@@ -108,64 +96,56 @@
      img.setAttribute("src", BAR_IMG1);
      wordCacheSpan.appendChild(img);
   }
-}

-function handleQueues(){
-    if(queuesRPC.readyState != 4){
-      return;
-   }
-   var queuesResponse = queuesRPC.responseXML;
-   //xml=getFirstChild(queuesResponse);
-   xml=getFirstChild(queuesResponse, "queues");
-   if(queuesResponse != null){
-      clearTable(document.getElementById("queueTable"), 1);
+
+   clearTable(document.getElementById("queueTable"), 1);
+
+   queuesTag = getFirstChild(crawlerTag, "queues");
+   indexingqueue=getFirstChild(queuesTag, "indexingqueue");
+   updateTable(indexingqueue, "indexing");
   
-      indexingqueue=getFirstChild(xml, "indexingqueue");
-      updateTable(indexingqueue, "indexing");
-      
-      indexingqueue_size=getValue(getFirstChild(indexingqueue, "size"));
-      indexingqueue_max=getValue(getFirstChild(indexingqueue, "max"));
-      document.getElementById("indexingqueuesize").firstChild.nodeValue=indexingqueue_size;
-      document.getElementById("indexingqueuemax").firstChild.nodeValue=indexingqueue_max;
-      
-      dbsize=getFirstChild(xml, "dbsize");
-      urlpublictextSize=getValue(getFirstChild(dbsize, "urlpublictext"));
-      rwipublictextSize=getValue(getFirstChild(dbsize, "rwipublictext"));
-      document.getElementById("urldbsize").firstChild.nodeValue=urlpublictextSize;
-      document.getElementById("rwidbsize").firstChild.nodeValue=rwipublictextSize;
-      
-      loaderqueue=getFirstChild(xml, "loaderqueue");
-      updateTable(loaderqueue, "loader");
-      
-      loaderqueue_size=getValue(getFirstChild(loaderqueue, "size"));
-      loaderqueue_max=getValue(getFirstChild(loaderqueue, "max"));
-      document.getElementById("loaderqueuesize").firstChild.nodeValue=loaderqueue_size;
-      document.getElementById("loaderqueuemax").firstChild.nodeValue=loaderqueue_max;
-      
-      localcrawlerqueue=getFirstChild(xml, "localcrawlerqueue");
-      localcrawlerqueue_size=getValue(getFirstChild(localcrawlerqueue, "size"));
-      localcrawlerqueue_state=getValue(getFirstChild(localcrawlerqueue, "state"));
-      document.getElementById("localcrawlerqueuesize").firstChild.nodeValue=localcrawlerqueue_size;
-      putQueueState("localcrawler", localcrawlerqueue_state);
-      
-      updateTable(localcrawlerqueue, "local crawler");
-      
-      limitcrawlerqueue=getFirstChild(xml, "limitcrawlerqueue");
-      updateTable(limitcrawlerqueue, "limitCrawlerTable");
-      limitcrawlerqueue_size=getValue(getFirstChild(limitcrawlerqueue, "size"));
-      limitcrawlerqueue_state=getValue(getFirstChild(limitcrawlerqueue, "state"));
-      document.getElementById("limitcrawlerqueuesize").firstChild.nodeValue=limitcrawlerqueue_size;
-      putQueueState("limitcrawler", limitcrawlerqueue_state);
-      updateTable(limitcrawlerqueue, "limit crawler");
-      
-      remotecrawlerqueue=getFirstChild(xml, "remotecrawlerqueue");
-      updateTable(remotecrawlerqueue, "remoteCrawlerTable");
-      remotecrawlerqueue_size=getValue(getFirstChild(remotecrawlerqueue, "size"));
-      remotecrawlerqueue_state=getValue(getFirstChild(remotecrawlerqueue, "state"));
-      document.getElementById("remotecrawlerqueuesize").firstChild.nodeValue=remotecrawlerqueue_size;
-      putQueueState("remotecrawler", remotecrawlerqueue_state);
-      updateTable(remotecrawlerqueue, "remote crawler");
-   }
+   indexingqueue_size=getValue(getFirstChild(indexingqueue, "size"));
+   indexingqueue_max=getValue(getFirstChild(indexingqueue, "max"));
+   document.getElementById("indexingqueuesize").firstChild.nodeValue=indexingqueue_size;
+   document.getElementById("indexingqueuemax").firstChild.nodeValue=indexingqueue_max;
+   
+   dbsize=getFirstChild(queuesTag, "dbsize");
+   urlpublictextSize=getValue(getFirstChild(dbsize, "urlpublictext"));
+   rwipublictextSize=getValue(getFirstChild(dbsize, "rwipublictext"));
+   document.getElementById("urldbsize").firstChild.nodeValue=urlpublictextSize;
+   document.getElementById("rwidbsize").firstChild.nodeValue=rwipublictextSize;
+   
+   loaderqueue=getFirstChild(queuesTag, "loaderqueue");
+   updateTable(loaderqueue, "loader");
+   
+   loaderqueue_size=getValue(getFirstChild(loaderqueue, "size"));
+   loaderqueue_max=getValue(getFirstChild(loaderqueue, "max"));
+   document.getElementById("loaderqueuesize").firstChild.nodeValue=loaderqueue_size;
+   document.getElementById("loaderqueuemax").firstChild.nodeValue=loaderqueue_max;
+   
+   localcrawlerqueue=getFirstChild(queuesTag, "localcrawlerqueue");
+   localcrawlerqueue_size=getValue(getFirstChild(localcrawlerqueue, "size"));
+   localcrawlerqueue_state=getValue(getFirstChild(localcrawlerqueue, "state"));
+   document.getElementById("localcrawlerqueuesize").firstChild.nodeValue=localcrawlerqueue_size;
+   putQueueState("localcrawler", localcrawlerqueue_state);
+   
+   updateTable(localcrawlerqueue, "local crawler");
+   
+   limitcrawlerqueue=getFirstChild(queuesTag, "limitcrawlerqueue");
+   updateTable(limitcrawlerqueue, "limitCrawlerTable");
+   limitcrawlerqueue_size=getValue(getFirstChild(limitcrawlerqueue, "size"));
+   limitcrawlerqueue_state=getValue(getFirstChild(limitcrawlerqueue, "state"));
+   document.getElementById("limitcrawlerqueuesize").firstChild.nodeValue=limitcrawlerqueue_size;
+   putQueueState("limitcrawler", limitcrawlerqueue_state);
+   updateTable(limitcrawlerqueue, "limit crawler");
+   
+   remotecrawlerqueue=getFirstChild(queuesTag, "remotecrawlerqueue");
+   updateTable(remotecrawlerqueue, "remoteCrawlerTable");
+   remotecrawlerqueue_size=getValue(getFirstChild(remotecrawlerqueue, "size"));
+   remotecrawlerqueue_state=getValue(getFirstChild(remotecrawlerqueue, "state"));
+   document.getElementById("remotecrawlerqueuesize").firstChild.nodeValue=remotecrawlerqueue_size;
+   putQueueState("remotecrawler", remotecrawlerqueue_state);
+   updateTable(remotecrawlerqueue, "remote crawler");
}

function putQueueState(queue, state) {
@@ -187,6 +167,8 @@
function updateTable(indexingqueue, tablename){
   indexingTable=document.getElementById("queueTable");
   entries=indexingqueue.getElementsByTagName("entry");
+   if (entries == null)
+       return;
         
     dark=false;
     for(i=0;i<entries.length;i++){
Index: htroot/js/Status.js
===================================================================
--- htroot/js/Status.js   (Revision 0)
+++ htroot/js/Status.js   (Revision 0)
@@ -0,0 +1,175 @@
+var statusRPC;
+var statusInterval = window.setInterval("updateStatus()", 30000);
+
+function updateStatus(){
+   statusRPC=createRequestObject();
+   statusRPC.open('get', '/xml/status_p.xml?html=');
+   statusRPC.onreadystatechange = handleStatus;
+   statusRPC.send(null);
+}
+
+function handleStatus(){
+    if(statusRPC.readyState != 4){
+      return;
+   }
+   var statusResponse = statusRPC.responseXML;
+   statusTag = getFirstChild(statusResponse, "status");
+   
+   document.getElementById("uptime").firstChild.nodeValue = uptimeFormat(getValue(getFirstChild(statusTag, "uptime")));
+   
+    var memoryTag = getFirstChild(statusTag, "memory");
+   document.getElementById("freeMemory").firstChild.nodeValue = byteFormat(getValue(getFirstChild(memoryTag, "free")));
+   document.getElementById("totalMemory").firstChild.nodeValue = byteFormat(getValue(getFirstChild(memoryTag, "total")));
+   document.getElementById("maxMemory").firstChild.nodeValue = byteFormat(getValue(getFirstChild(memoryTag, "max")));
+   
+   
+    var trafficTag = getFirstChild(statusTag, "traffic");
+    document.getElementById("trafficProxy").firstChild.nodeValue = byteFormat(getValue(getFirstChild(trafficTag, "proxy")));
+    document.getElementById("trafficCrawler").firstChild.nodeValue = byteFormat(getValue(getFirstChild(trafficTag, "crawler")));
+   
+   var connectionsTag = getFirstChild(statusTag, "connections");
+    document.getElementById("connectionsActive").firstChild.nodeValue = getValue(getFirstChild(connectionsTag, "active"));
+    document.getElementById("connectionsMax").firstChild.nodeValue = getValue(getFirstChild(connectionsTag, "max"));
+   
+   var queuesTag = getFirstChild(statusTag, "queues");
+    document.getElementById("indexingQueueSize").firstChild.nodeValue = getValue(getFirstChild(queuesTag, "indexingSize"));
+    document.getElementById("indexingQueueMax").firstChild.nodeValue = getValue(getFirstChild(queuesTag, "indexingMax"));
+    document.getElementById("indexingQueuePercent").style.width = getValue(getFirstChild(queuesTag, "indexingPercent")).toString() + "%";
+
+    document.getElementById("loaderQueueSize").firstChild.nodeValue = getValue(getFirstChild(queuesTag, "loaderSize"));
+    document.getElementById("loaderQueueMax").firstChild.nodeValue = getValue(getFirstChild(queuesTag, "loaderMax"));
+    document.getElementById("loaderQueuePercent").style.width = getValue(getFirstChild(queuesTag, "loaderPercent")).toString() + "%";
+
+    document.getElementById("localCrawlQueueSize").firstChild.nodeValue = getValue(getFirstChild(queuesTag, "localCrawlSize"));
+   var localcrawlerqueuepaused = getValue(getFirstChild(queuesTag, "localCrawlPaused"));
+   putQueueState("localcrawler", localcrawlerqueuepaused);
+
+    document.getElementById("remoteTriggeredCrawlQueueSize").firstChild.nodeValue = getValue(getFirstChild(queuesTag, "remoteTriggeredCrawlSize"));
+   var remotecrawlerqueuepaused = getValue(getFirstChild(queuesTag, "remoteTriggeredCrawlPaused"));
+   putQueueState("remotecrawler", remotecrawlerqueuepaused);
+   
+    document.getElementById("stackCrawlQueueSize").firstChild.nodeValue = getValue(getFirstChild(queuesTag, "stackCrawlSize"));
+   
+
+   
+    var warningsTag = getFirstChild(statusTag, "warnings");
+    if (getValue(getFirstChild(warningsTag, "diskSpaceLow")) == 1){
+        var diskminspace = getValue(getFirstChild(queuesTag, "diskSpaceLow"));
+        var msg = "Free disk space is lower than " +
+            diskminspace + "Crawling has been disabled. Please fix it as soon as possible and restart YaCy.";
+        putMessage(msg, "bad", "warningDiskSpaceLow");

+    } else removeMessage("warningDiskSpaceLow");
+
+
+    if (getValue(getFirstChild(warningsTag, "peerStatus")) == "junior"){
+        var msg = "You cannot be reached from outside. A possible reason is that you are behind a firewall, NAT or Router. " +

+            "But you can search the internet using the other peers' global index on your own search page. We encourage you " +
+            "to open your firewall for the port you configured (usually: 8080), or to set up a 'virtual server' in your " +
+            "router settings (often called DMZ). Please be fair, contribute your own index to the global index.";
+        putMessage(msg, "bad", "warningStatusJunior");
+    } else removeMessage("warningStatusJunior");
+   
+    if (getValue(getFirstChild(warningsTag, "peerStatus")) == "virgin"){
+        var msg = "You have not published your peer seed yet. This happens automatically, just wait.";
+        putMessage(msg, "bad", "urgentStatusVirgin");
+    } else removeMessage("urgentStatusVirgin");
+
+   
+    if (getValue(getFirstChild(warningsTag, "goOnline")) == "1"){
+        var msg = "The peer must go online to get a peer address.";
+        putMessage(msg, "bad", "warningGoOnline");
+    } else removeMessage("warningGoOnline");
+}   
+
+function putMessage(msg, type, parentId){
+    var parent = document.getElementById(parentId);
+    if (parent == null)
+        return;
+    removeAllChildren(parent);
+   
+    var dt = document.createElement("dt");
+    nodeSetAttribute(dt, "class", "hintIcon");
+    parent.appendChild(dt);
+   
+    var img = document.createElement("img");
+    nodeSetAttribute(img, "width", 32);
+    nodeSetAttribute(img, "height", 32);
+    nodeSetAttribute(img, "src", "env/grafics/" + type + ".png");
+    nodeSetAttribute(img, "alt", type);
+    dt.appendChild(img);
+   
+    var text = document.createTextNode(msg);
+    var dd = document.createElement("dd");
+    nodeSetAttribute(dd, "class", "hint");
+    dd.appendChild(text);
+    parent.appendChild(dd);
+    parent.appendChild(document.createElement("br"));
+}
+
+function nodeSetAttribute(node, attribute, value){
+    var atr = document.createAttribute(attribute);
+    atr.nodeValue = value;
+    node.setAttributeNode(atr);
+}
+
+function removeMessage(parentId){
+    var parent = document.getElementById(parentId);
+    if (parent == null)
+        return;
+    removeAllChildren(parent);
+}
+
+function putQueueState(queue, paused) {
+   a = document.getElementById(queue + "stateA");
+   img = document.getElementById(queue + "stateIMG");
+   td = document.getElementById(queue + "stateTD");
+   removeAllChildren(td);
+   
+   if (paused == 1) {
+      a.href = "Status.html?continue=" + queue;
+      a.title = "Continue this queue";
+      img.src = "/env/grafics/start.gif";
+      img.alt = "Continue this queue";
+        var text = document.createTextNode("(paused)");
+        td.appendChild(text);
+   } else {
+      a.href = "Status.html?pause=" + queue;
+      a.title = "Pause this queue";
+      img.src = "/env/grafics/stop.gif";
+      img.alt = "Pause this queue";
+   }
+}
+
+function uptimeFormat(uptime){
+    var mins = uptime % 60;
+    var hours = Math.floor((uptime % 1440) / 60);
+    var days = Math.floor(uptime / 1440);
+
+    var span = days > 1 ? " days " : " day ";
+    var h = hours < 10 ? "0" + hours.toString() : hours.toString();
+    var m = mins < 10 ? "0" + mins.toString() : mins.toString();
+   
+    return days.toString() + span + h + ":" + m;
+}
+
+function byteFormat(byteCount){
+    var val;
+    var suffix;
+    var bc = byteCount.replace(/[,. ]/g, "");
+   
+    if (bc > 1073741824) {
+        val = (bc / 1073741824).toFixed(2);
+        suffix = " GB";

+    } else if (bc > 1048576) {

+        val = (bc / 1048576).toFixed(2);

+        suffix = " MB";

+    } else if (bc > 1024) {

+        val = (bc / 1024).toFixed(2);

+       suffix = " KB";

+    } else {

+        val = bc;

+        suffix = " Bytes";
+    }

+    return val + suffix;
+}
+
Index: htroot/WatchCrawler_p.java
===================================================================
--- htroot/WatchCrawler_p.java   (Revision 4957)
+++ htroot/WatchCrawler_p.java   (Arbeitskopie)
@@ -35,7 +35,7 @@
import java.util.regex.Pattern;
import java.util.regex.PatternSyntaxException;

-import xml.queues_p;
+import xml.crawler_p;
import de.anomic.crawler.CrawlEntry;
import de.anomic.crawler.CrawlProfile;
import de.anomic.crawler.SitemapImporter;
@@ -64,7 +64,7 @@
         // return variable that accumulates replacements
         plasmaSwitchboard sb = (plasmaSwitchboard) env;
         // inital values for AJAX Elements (without JavaScript)
-        serverObjects prop = queues_p.respond(header, post, env);
+        serverObjects prop = crawler_p.respond(header, post, env);
         prop.put("forwardToCrawlStart", "0");
         
         prop.put("info", "0");
Index: htroot/Status_p.inc
===================================================================
--- htroot/Status_p.inc   (Revision 4957)
+++ htroot/Status_p.inc   (Arbeitskopie)
@@ -8,7 +8,7 @@
     #(peerStatistics)#

       Unknown

       ::

-      <div>Uptime: #[uptime]#</div>

+      <div>Uptime: <span id="uptime">#[uptime]#</span></div>

       #(/peerStatistics)#

       </dd>

     

@@ -50,47 +50,47 @@
       

     <dt><a href="PerformanceMemory_p.html">Memory Usage</a></dt>

     <dd>

-    free: #[freeMemory]#<br/>

-    total: #[totalMemory]#<br/>

-    max: #[maxMemory]#</dd>

+    free: <span id="freeMemory">#[freeMemory]#</span><br/>

+    total: <span id="totalMemory">#[totalMemory]#</span><br/>

+    max: <span id="maxMemory">#[maxMemory]#</span></dd>

     

     <dt>Traffic [<a href="Status.html?ResetTraffic=">Reset</a>]</dt>

-    <dd>Proxy: #[trafficProxy]#<br/>Crawler: #[trafficCrawler]#</dd>

+    <dd>Proxy: <span id="trafficProxy">#[trafficProxy]#</span><br/>Crawler: <span id="trafficCrawler">#[trafficCrawler]#</span></dd>

     

     <dt><a href="Connections_p.html">Incoming Connections</a></dt>

-    <dd>Active: #[connectionsActive]# | Max: #[connectionsMax]#</dd>

+    <dd>Active: <span id="connectionsActive">#[connectionsActive]#</span> | Max: <span id="connectionsMax">#[connectionsMax]#</span></dd>

     

     <dt>Queues</dt>

     <dd>

     <a href="IndexCreateIndexingQueue_p.html">Indexing Queue</a>:

     <table>

       <tr>

-      <td><div class="ProgressBar"><div class="ProgressBarFill" style="width:#[indexingQueuePercent]#%;" ></div></div></td>

-      <td><div class="ProgressBarCaption">#[indexingQueueSize]#&nbsp;|&nbsp;#[indexingQueueMax]#</div></td>

+      <td><div class="ProgressBar"><div id="indexingQueuePercent" class="ProgressBarFill" style="width:#[indexingQueuePercent]#%;" ></div></div></td>

+      <td><div class="ProgressBarCaption"><span id="indexingQueueSize">#[indexingQueueSize]#</span>&nbsp;|&nbsp;<span id="indexingQueueMax">#[indexingQueueMax]#</span></div></td>

       </tr>

     </table><br />

     

     <a href="IndexCreateLoaderQueue_p.html">Loader Queue</a>:

     <table>

       <tr>

-        <td><div class="ProgressBar"><div class="ProgressBarFill" style="width:#[loaderQueuePercent]#%;" ></div></div></td>

-        <td><div class="ProgressBarCaption">#[loaderQueueSize]#&nbsp;|&nbsp;#[loaderQueueMax]#</div></td>

+        <td><div class="ProgressBar"><div id="loaderQueuePercent" class="ProgressBarFill" style="width:#[loaderQueuePercent]#%;" ></div></div></td>

+        <td><div class="ProgressBarCaption"><span id="loaderQueueSize">#[loaderQueueSize]#</span>&nbsp;|&nbsp;<span id="loaderQueueMax">#[loaderQueueMax]#</span></div></td>

       </tr>

       <tr>

         <td><a href="IndexCreateWWWLocalQueue_p.html">Local Crawl</a></td>

-        <td>#[localCrawlQueueSize]#</td>

-        <td><a href="Status.html?#(localCrawlPaused)#pauseCrawlJob::continueCrawlJob#(/localCrawlPaused)#=&amp;jobType=localCrawl" title="#(localCrawlPaused)#pause local crawl::continue local crawl#(/localCrawlPaused)#"><img src="env/grafics/#(localCrawlPaused)#stop.gif::start.gif#(/localCrawlPaused)#" alt="#(localCrawlPaused)#pause local crawl::continue local crawl#(/localCrawlPaused)#" style="width:12px;height:12px;" /></a></td>

-        <td>#(localCrawlPaused)#&nbsp;::(paused)#(/localCrawlPaused)#</td>

+        <td><span id="localCrawlQueueSize">#[localCrawlQueueSize]#</span></td>

+        <td><a id="localcrawlerstateA" href="Status.html?#(localCrawlPaused)#pauseCrawlJob::continueCrawlJob#(/localCrawlPaused)#=&amp;jobType=localCrawl" title="#(localCrawlPaused)#pause local crawl::continue local crawl#(/localCrawlPaused)#"><img id="localcrawlerstateIMG" src="env/grafics/#(localCrawlPaused)#stop.gif::start.gif#(/localCrawlPaused)#" alt="#(localCrawlPaused)#pause local crawl::continue local crawl#(/localCrawlPaused)#" style="width:12px;height:12px;" /></a></td>

+        <td id="localcrawlerstateTD">#(localCrawlPaused)#&nbsp;::(paused)#(/localCrawlPaused)#</td>

       </tr>

       <tr>

         <td><a href="IndexCreateWWWRemoteQueue_p.html">Remote triggered Crawl</a></td>

-        <td>#[remoteTriggeredCrawlQueueSize]#</td>

-        <td><a href="Status.html?#(remoteTriggeredCrawlPaused)#pauseCrawlJob::continueCrawlJob#(/remoteTriggeredCrawlPaused)#=&amp;jobType=remoteTriggeredCrawl" title="#(remoteTriggeredCrawlPaused)#pause remote triggered crawl::continue remote triggered crawl#(/remoteTriggeredCrawlPaused)#"><img src="env/grafics/#(remoteTriggeredCrawlPaused)#stop.gif::start.gif#(/remoteTriggeredCrawlPaused)#" alt="#(remoteTriggeredCrawlPaused)#pause remote triggered crawl::continue remote triggered crawl#(/remoteTriggeredCrawlPaused)#" style="width:12px;height:12px;" /></a></td>

-        <td>#(remoteTriggeredCrawlPaused)#&nbsp;::(paused)#(/remoteTriggeredCrawlPaused)#</td>

+        <td><span id="remoteTriggeredCrawlQueueSize">#[remoteTriggeredCrawlQueueSize]#</span></td>

+        <td><a id="remotecrawlerstateA" href="Status.html?#(remoteTriggeredCrawlPaused)#pauseCrawlJob::continueCrawlJob#(/remoteTriggeredCrawlPaused)#=&amp;jobType=remoteTriggeredCrawl" title="#(remoteTriggeredCrawlPaused)#pause remote triggered crawl::continue remote triggered crawl#(/remoteTriggeredCrawlPaused)#"><img id="remotecrawlerstateIMG" src="env/grafics/#(remoteTriggeredCrawlPaused)#stop.gif::start.gif#(/remoteTriggeredCrawlPaused)#" alt="#(remoteTriggeredCrawlPaused)#pause remote triggered crawl::continue remote triggered crawl#(/remoteTriggeredCrawlPaused)#" style="width:12px;height:12px;" /></a></td>

+        <td id="remotecrawlerstateTD">#(remoteTriggeredCrawlPaused)#&nbsp;::(paused)#(/remoteTriggeredCrawlPaused)#</td>

       </tr>

       <tr>

         <td>Pre-Queueing</td>

-        <td>#[stackCrawlQueueSize]#</td>

+        <td><span id="stackCrawlQueueSize">#[stackCrawlQueueSize]#<span></td>

         <td></td>

         <td></td>

       </tr> 

@@ -113,4 +113,4 @@
     

   </dl>

</fieldset>

-</div>
\ No newline at end of file
+</div>
Index: htroot/Status.html
===================================================================
--- htroot/Status.html   (Revision 4957)
+++ htroot/Status.html   (Arbeitskopie)
@@ -3,6 +3,11 @@
   <head>

     <title>YaCy '#[clientname]#': Console Status</title>

     #%env/templates/metas.template%#

+    #(updateStatusTable)#::

+    <script type="text/javascript" src="/js/ajax.js"></script>

+    <script type="text/javascript" src="/js/xml.js"></script>

+    <script type="text/javascript" src="/js/Status.js"></script>

+    #(/updateStatusTable)#

     <script type="text/javascript">

       function loadBanner() {

         for (var i = 0; i < document.images.length; i++) {

@@ -53,22 +58,30 @@
     Please open the <a href="ConfigAccounts_p.html">accounts configuration</a> page <strong>immediately</strong>

     and set an administration password.

     </dd>

+    <br />

     #(/urgentSetPassword)#



+    <span id="urgentStatusVirgin">

   #(urgentStatusVirgin)#::

     <dt class="hintIcon"><img src="env/grafics/bad.png" width="32" height="32" alt="bad"/></dt>

     <dd class="hint">You have not published your peer seed yet. This happens automatically, just wait.

     </dd>

+    <br />

     #(/urgentStatusVirgin)#

+    </span>

   

     <!-- warnings -->

     

+    <span id="warningGoOnline">

     #(warningGoOnline)#::

     <dt class="hintIcon"><img src="env/grafics/bad.png" width="32" height="32" alt="bad"/></dt>

     <dd class="hint">The peer must go online to get a peer address.

     </dd>

+    <br />

     #(/warningGoOnline)#

+    </span>

     

+    <span id="warningStatusJunior">

     #(warningStatusJunior)#::

     <dt class="hintIcon"><img src="env/grafics/bad.png" width="32" height="32" alt="bad"/></dt>

     <dd class="hint">You cannot be reached from outside.

@@ -79,14 +92,19 @@
     or to set up a 'virtual server' in your router settings (often called DMZ).

     Please be fair, contribute your own index to the global index.

     </dd>

+    <br />

     #(/warningStatusJunior)#

+    </span>

     

+    <span id="warningDiskSpaceLow">

     #(warningDiskSpaceLow)#::

     <dt class="hintIcon"><img src="env/grafics/bad.png" width="32" height="32" alt="bad"/></dt>

     <dd class="hint">Free disk space is lower than #[minSpace]#. Crawling has been disabled. Please fix

     it as soon as possible and restart YaCy.

     </dd>

+    <br />

     #(/warningDiskSpaceLow)#

+    </span>

     

      <!-- hints -->

     

Index: htroot/Status.java
===================================================================
--- htroot/Status.java   (Revision 4957)
+++ htroot/Status.java   (Arbeitskopie)
@@ -121,10 +121,14 @@
         sb.updateMySeed();



         boolean adminaccess = sb.adminAuthenticated(header) >= 2;

+

+

         if (adminaccess) {

+            prop.put("updateStatusTable", "1");

             prop.put("showPrivateTable", "1");

             prop.put("privateStatusTable", "Status_p.inc");

         } else {

+            prop.put("updateStatusTable", "0");

             prop.put("showPrivateTable", "0");

             prop.put("privateStatusTable", "");

         }

datengrab
 
Beiträge: 366
Registriert: Sa Aug 25, 2007 7:22 pm

Re: Status autoupdate

Beitragvon datengrab » Do Jun 26, 2008 2:59 pm

Das das Forum meint, das der Beitrag zu groß wäre kommt hier der zweite Teil :mrgreen:

Code: Alles auswählen
Index: htroot/xml/status_p.java
===================================================================
--- htroot/xml/status_p.java   (Revision 4957)
+++ htroot/xml/status_p.java   (Arbeitskopie)
@@ -1,4 +1,4 @@
-// /xml.queues/status_p.java

+// /xml/status_p.java

// -------------------------------

// part of the yacy

//

@@ -42,11 +42,14 @@
import de.anomic.http.httpHeader;

import de.anomic.http.httpdByteCountInputStream;

import de.anomic.http.httpdByteCountOutputStream;

-import de.anomic.plasma.plasmaSwitchboard;

+import de.anomic.plasma.plasmaSwitchboard;
+import de.anomic.server.serverCore;
+import de.anomic.server.serverDate;

import de.anomic.server.serverMemory;

import de.anomic.server.serverObjects;

import de.anomic.server.serverProcessor;

import de.anomic.server.serverSwitch;

+import de.anomic.yacy.yacySeed;


public class status_p {

     

@@ -60,17 +63,17 @@
         prop.put("rejected", "0");

         sb.updateMySeed();

         final int  cacheOutSize = sb.webIndex.dhtOutCacheSize();

-        final long cacheMaxSize = sb.getConfigLong(plasmaSwitchboard.WORDCACHE_MAX_COUNT, 10000);

+        final long cacheMaxSize = sb.getConfigLong(plasmaSwitchboard.WORDCACHE_MAX_COUNT, 10000);
+        final long uptime = 60000 * Long.parseLong(sb.webIndex.seedDB.mySeed().get(yacySeed.UPTIME, "0"));

+
+
+        ///////////////
+        //  sidebar  //
+        ///////////////
+       
+        prop.put("uptime", serverDate.formatInterval(uptime));

         prop.putNum("ppm", sb.webIndex.seedDB.mySeed().getPPM());

-        prop.putNum("qpm", sb.webIndex.seedDB.mySeed().getQPM());

-        prop.putNum("wordCacheSize", sb.webIndex.dhtOutCacheSize() + sb.webIndex.dhtInCacheSize());

-        prop.putNum("wordCacheWSize", cacheOutSize);

-        prop.putNum("wordCacheKSize", sb.webIndex.dhtInCacheSize());

-        prop.putNum("wordCacheMaxSize", cacheMaxSize);

-        prop.put("wordCacheWCount", cacheOutSize);

-        prop.put("wordCacheMaxCount", cacheMaxSize);



-      //

      // memory usage and system attributes

         prop.putNum("freeMemory", serverMemory.free());

         prop.putNum("totalMemory", serverMemory.total());

@@ -81,7 +84,66 @@
      prop.put("trafficIn", httpdByteCountInputStream.getGlobalCount());

      prop.put("trafficProxy", httpdByteCountOutputStream.getAccountCount("PROXY"));

      prop.put("trafficCrawler", httpdByteCountInputStream.getAccountCount("CRAWLER"));

+
+        // connection information
+        serverCore httpd = (serverCore) sb.getThread("10_httpd");

+        prop.putNum("connectionsActive", httpd.getJobCount());

+        prop.putNum("connectionsMax", httpd.getMaxSessionCount());

+
+        // Queue information

+        int indexingJobCount = sb.getThread("80_indexing").getJobCount() + sb.webIndex.queuePreStack.getActiveQueueSize();

+        int indexingMaxCount = (int) sb.getConfigLong(plasmaSwitchboard.INDEXER_SLOTS, 30);

+        int indexingPercent = (indexingMaxCount == 0) ? 0 : indexingJobCount * 100 / indexingMaxCount;

+        prop.putNum("indexingQueueSize", indexingJobCount);

+        prop.putNum("indexingQueueMax", indexingMaxCount);

+        prop.put("indexingQueuePercent",(indexingPercent>100) ? 100 : indexingPercent);

+
+        int loaderJobCount = sb.crawlQueues.size();

+        int loaderMaxCount = Integer.parseInt(sb.getConfig(plasmaSwitchboard.CRAWLER_THREADS_ACTIVE_MAX, "10"));

+        int loaderPercent = (loaderMaxCount==0)?0:loaderJobCount*100/loaderMaxCount;

+        prop.putNum("loaderQueueSize", loaderJobCount);

+        prop.putNum("loaderQueueMax", loaderMaxCount);       

+        prop.put("loaderQueuePercent", (loaderPercent>100) ? 100 : loaderPercent);

+
+        prop.putNum("localCrawlQueueSize", sb.getThread(plasmaSwitchboard.CRAWLJOB_LOCAL_CRAWL).getJobCount());

+        prop.put("localCrawlPaused",sb.crawlJobIsPaused(plasmaSwitchboard.CRAWLJOB_LOCAL_CRAWL) ? "1" : "0");



+        prop.putNum("remoteTriggeredCrawlQueueSize", sb.getThread(plasmaSwitchboard.CRAWLJOB_REMOTE_TRIGGERED_CRAWL).getJobCount());

+        prop.put("remoteTriggeredCrawlPaused",sb.crawlJobIsPaused(plasmaSwitchboard.CRAWLJOB_REMOTE_TRIGGERED_CRAWL) ? "1" : "0");

+
+        prop.putNum("stackCrawlQueueSize", sb.crawlStacker.size());
+     
+     
+        ////////////////////////
+        //  infos / warnings  //
+        ////////////////////////
+       
+        // free disk space

+        if (!sb.observer.getDisksOK())

+        {

+            String minFree = serverMemory.bytesToString(sb.observer.getMinFreeDiskSpace());

+            prop.put("warningDiskSpaceLow", "1");

+            prop.put("warningDiskSpaceMin", minFree);

+        }
+        else
+            prop.put("warningDiskSpaceLow", "0");

+       
+        yacySeed thisSeed = sb.webIndex.seedDB.mySeed();
+        // peer status type       
+        final String peerStatus = (thisSeed == null ? yacySeed.PEERTYPE_VIRGIN : thisSeed.get(yacySeed.PEERTYPE, yacySeed.PEERTYPE_VIRGIN));

+        if (peerStatus.equals(yacySeed.PEERTYPE_VIRGIN))

+            prop.put("warningPeerStatus", "virgin");
+        else if (peerStatus.equals(yacySeed.PEERTYPE_JUNIOR))

+            prop.put("warningPeerStatus", "junior");
+        else prop.put("warningPeerStatus", "");
+       
+        if (thisSeed.getPublicAddress() == null){
+            if (sb.webIndex.seedDB.mySeed().getPublicAddress() == null)

+                prop.put("warningGoOnline", "1");
+            else
+                prop.put("warningGoOnline", "0");
+        }
+

         // return rewrite properties

         return prop;

     }

Index: htroot/xml/queues_p.xml
===================================================================
--- htroot/xml/queues_p.xml   (Revision 4957)
+++ htroot/xml/queues_p.xml   (Arbeitskopie)
@@ -1,84 +0,0 @@
-<?xml version="1.0"?>
-<queues>
-<dbsize>
-  <urlpublictext>#[urlpublictextSize]#</urlpublictext>
-  <rwipublictext>#[rwipublictextSize]#</rwipublictext>
-</dbsize>
-<indexingqueue>
-  <size>#[indexingSize]#</size>
-  <max>#[indexingMax]#</max>
-#{list-indexing}#
-  <entry>
-    <profile>#[profile]#</profile>
-    <initiator>#[initiator]#</initiator>
-    <depth>#[depth]#</depth>
-    <modified>#[modified]#</modified>
-    <anchor>#[anchor]#</anchor>
-    <url>#[url]#</url>
-    <size>#[size]#</size>
-    <hash>#[hash]#</hash>
-    <inProcess>#(inProcess)#false::true#(/inProcess)#</inProcess>
-  </entry>
-#{/list-indexing}#
-</indexingqueue>
-<loaderqueue>
-  <size>#[loaderSize]#</size>
-  <max>#[loaderMax]#</max>
-#{list-loader}#
-  <entry>
-    <profile>#[profile]#</profile>
-    <initiator>#[initiator]#</initiator>
-    <depth>#[depth]#</depth>
-    <url>#[url]#</url>
-  </entry>
-#{/list-loader}#
-</loaderqueue>
-<localcrawlerqueue>
-  <size>#[localCrawlSize]#</size>
-  <state>#[localCrawlState]#</state>
-#{list-local}#
-  <entry>
-    <profile>#[profile]#</profile>
-    <initiator>#[initiator]#</initiator>
-    <depth>#[depth]#</depth>
-    <modified>#[modified]#</modified>
-    <anchor>#[anchor]#</anchor>
-    <url>#[url]#</url>
-    <hash>#[hash]#</hash>
-    <inProcess>#(inProcess)#false::true#(/inProcess)#</inProcess>
-  </entry>
-#{/list-local}#
-</localcrawlerqueue>
-<limitcrawlerqueue>
-  <size>#[limitCrawlSize]#</size>
-  <state>#[limitCrawlState]#</state>
-#{list-limit}#
-  <entry>
-    <profile>#[profile]#</profile>
-   <initiator>#[initiator]#</initiator>
-    <depth>#[depth]#</depth>
-    <modified>#[modified]#</modified>
-    <anchor>#[anchor]#</anchor>
-    <url>#[url]#</url>
-    <hash>#[hash]#</hash>
-    <inProcess>#(inProcess)#false::true#(/inProcess)#</inProcess>
-  </entry>
-#{/list-limit}#
-</limitcrawlerqueue>
-<remotecrawlerqueue>
-  <size>#[remoteCrawlSize]#</size>
-  <state>#[remoteCrawlState]#</state>
-#{list-remote}#
-  <entry>
-    <profile>#[profile]#</profile>
-   <initiator>#[initiator]#</initiator>
-    <depth>#[depth]#</depth>
-    <modified>#[modified]#</modified>
-    <anchor>#[anchor]#</anchor>
-    <url>#[url]#</url>
-    <hash>#[hash]#</hash>
-    <inProcess>#(inProcess)#false::true#(/inProcess)#</inProcess>
-  </entry>
-#{/list-remote}#
-</remotecrawlerqueue>
-</queues>
Index: htroot/xml/crawler_p.java
===================================================================
--- htroot/xml/crawler_p.java   (Revision 4957)
+++ htroot/xml/crawler_p.java   (Arbeitskopie)
@@ -1,4 +1,4 @@
-// /xml.queues/indexing_p.java

+// /xml/crawler_p.java

// -------------------------------

// part of the AnomicHTTPD caching proxy

// (C) by Michael Peter Christen; mc@anomic.de

@@ -40,11 +40,6 @@
// the intact and unchanged copyright notice.

// Contributions and changes to the program code must be marked as such.



-// You must compile this file with

-// javac -classpath .:../classes IndexCreate_p.java

-// if the shell's current path is HTROOT

-

-//package xml.queues;

package xml;



import java.text.SimpleDateFormat;

@@ -57,12 +52,16 @@
import de.anomic.crawler.IndexingStack;

import de.anomic.crawler.NoticedURL;

import de.anomic.http.httpHeader;

+import de.anomic.http.httpdByteCountInputStream;

+import de.anomic.http.httpdByteCountOutputStream;

import de.anomic.plasma.plasmaSwitchboard;

+import de.anomic.server.serverMemory;

import de.anomic.server.serverObjects;

+import de.anomic.server.serverProcessor;

import de.anomic.server.serverSwitch;

import de.anomic.yacy.yacySeed;

-

-public class queues_p {

+
+public class crawler_p {

     

     public static final String STATE_RUNNING = "running";

     public static final String STATE_PAUSED = "paused";

@@ -84,7 +83,23 @@
         //int showRejectedCount = 10;

         

         yacySeed initiator;

-       

+
+
+        // return variable that accumulates replacements

+        sb.updateMySeed();

+        final int  cacheOutSize = sb.webIndex.dhtOutCacheSize();

+        final long cacheMaxSize = sb.getConfigLong(plasmaSwitchboard.WORDCACHE_MAX_COUNT, 10000);

+        prop.putNum("ppm", sb.webIndex.seedDB.mySeed().getPPM());

+        prop.putNum("wordCacheWSize", cacheOutSize);

+        prop.putNum("wordCacheMaxSize", cacheMaxSize);

+        prop.put("wordCacheWCount", cacheOutSize);

+        prop.put("wordCacheMaxCount", cacheMaxSize);

+

+      // traffic

+      prop.put("trafficCrawler", httpdByteCountInputStream.getAccountCount("CRAWLER"));

+

+
+

         //indexing queue
         prop.putNum("indexingSize", sb.getThread(plasmaSwitchboard.INDEXER).getJobCount() + sb.webIndex.queuePreStack.getActiveQueueSize());

         prop.putNum("indexingMax", (int) sb.getConfigLong(plasmaSwitchboard.INDEXER_SLOTS, 30));

Index: htroot/xml/status_p.xml
===================================================================
--- htroot/xml/status_p.xml   (Revision 4957)
+++ htroot/xml/status_p.xml   (Arbeitskopie)
@@ -1,13 +1,7 @@
<?xml version="1.0"?>
<status>
+  <uptime>#[uptime]#</uptime>
   <ppm>#[ppm]#</ppm>
-  <wordCacheSize>#[wordCacheSize]#</wordCacheSize>
-  <wordCacheWSize>#[wordCacheWSize]#</wordCacheWSize>
-  <wordCacheWCount>#[wordCacheWCount]#</wordCacheWCount>
-  <wordCacheKSize>#[wordCacheKSize]#</wordCacheKSize>
-  <wordCacheMaxSize>#[wordCacheMaxSize]#</wordCacheMaxSize>
-  <wordCacheMaxCount>#[wordCacheMaxCount]#</wordCacheMaxCount>
-
   <memory>
     <free>#[freeMemory]#</free>
     <total>#[totalMemory]#</total>
@@ -19,4 +13,27 @@
   <proxy>#[trafficProxy]#</proxy>
   <crawler>#[trafficCrawler]#</crawler>
   </traffic>
+  <connections>
+    <active>#[connectionsActive]#</active>

+    <max>#[connectionsMax]#</max>
+  </connections>
+  <queues>
+    <indexingSize>#[indexingQueueSize]#</indexingSize>

+    <indexingMax>#[indexingQueueMax]#</indexingMax>

+    <indexingPercent>#[indexingQueuePercent]#</indexingPercent>
+    <loaderSize>#[loaderQueueSize]#</loaderSize>

+    <loaderMax>#[loaderQueueMax]#</loaderMax>

+    <loaderPercent>#[loaderQueuePercent]#</loaderPercent>
+    <localCrawlSize>#[localCrawlQueueSize]#</localCrawlSize>
+    <localCrawlPaused>#[localCrawlPaused]#</localCrawlPaused>
+    <remoteTriggeredCrawlSize>#[remoteTriggeredCrawlQueueSize]#</remoteTriggeredCrawlSize>
+    <remoteTriggeredCrawlPaused>#[remoteTriggeredCrawlPaused]#</remoteTriggeredCrawlPaused>
+    <stackCrawlSize>#[stackCrawlQueueSize]#</stackCrawlSize>
+  </queues>
+  <warnings>
+    <diskSpaceLow>#[warningDiskSpaceLow]#</diskSpaceLow>
+    <diskSpaceMin>#[warningDiskSpaceMin]#</diskSpaceMin>
+    <peerStatus>#[warningPeerStatus]#</peerStatus>
+    <goOnline>#[warningGoOnline]#</goOnline>
+  </warnings>
</status>
Index: htroot/xml/queues_p.java
===================================================================
--- htroot/xml/queues_p.java   (Revision 4957)
+++ htroot/xml/queues_p.java   (Arbeitskopie)
@@ -1,206 +0,0 @@
-// /xml.queues/indexing_p.java

-// -------------------------------

-// part of the AnomicHTTPD caching proxy

-// (C) by Michael Peter Christen; mc@anomic.de

-// first published on http://www.anomic.de

-// Frankfurt, Germany, 2004, 2005

-// last major change: 28.10.2005

-// this file is contributed by Alexander Schier

-//

-// This program is free software; you can redistribute it and/or modify

-// it under the terms of the GNU General Public License as published by

-// the Free Software Foundation; either version 2 of the License, or

-// (at your option) any later version.

-//

-// This program is distributed in the hope that it will be useful,

-// but WITHOUT ANY WARRANTY; without even the implied warranty of

-// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the

-// GNU General Public License for more details.

-//

-// You should have received a copy of the GNU General Public License

-// along with this program; if not, write to the Free Software

-// Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA  02111-1307  USA

-//

-// Using this software in any meaning (reading, learning, copying, compiling,

-// running) means that you agree that the Author(s) is (are) not responsible

-// for cost, loss of data or any harm that may be caused directly or indirectly

-// by usage of this softare or this documentation. The usage of this software

-// is on your own risk. The installation and usage (starting/running) of this

-// software may allow other people or application to access your computer and

-// any attached devices and is highly dependent on the configuration of the

-// software which must be done by the user of the software; the author(s) is

-// (are) also not responsible for proper configuration and usage of the

-// software, even if provoked by documentation provided together with

-// the software.

-//

-// Any changes to this file according to the GPL as documented in the file

-// gpl.txt aside this file in the shipment you received can be done to the

-// lines that follows this copyright notice here, but changes must not be

-// done inside the copyright notive above. A re-distribution must contain

-// the intact and unchanged copyright notice.

-// Contributions and changes to the program code must be marked as such.

-

-// You must compile this file with

-// javac -classpath .:../classes IndexCreate_p.java

-// if the shell's current path is HTROOT

-

-//package xml.queues;

-package xml;

-

-import java.text.SimpleDateFormat;

-import java.util.ArrayList;

-import java.util.Date;

-import java.util.Iterator;

-import java.util.Locale;

-

-import de.anomic.crawler.CrawlEntry;

-import de.anomic.crawler.IndexingStack;

-import de.anomic.crawler.NoticedURL;

-import de.anomic.http.httpHeader;

-import de.anomic.plasma.plasmaSwitchboard;

-import de.anomic.server.serverObjects;

-import de.anomic.server.serverSwitch;

-import de.anomic.yacy.yacySeed;

-

-public class queues_p {

-   

-    public static final String STATE_RUNNING = "running";

-    public static final String STATE_PAUSED = "paused";

-   

-    private static SimpleDateFormat dayFormatter = new SimpleDateFormat("yyyy/MM/dd", Locale.US);

-    private static String daydate(Date date) {

-        if (date == null) return "";

-        return dayFormatter.format(date);

-    }

-   

-    public static serverObjects respond(httpHeader header, serverObjects post, serverSwitch<?> env) {

-        // return variable that accumulates replacements

-        plasmaSwitchboard sb = (plasmaSwitchboard) env;

-        //wikiCode wikiTransformer = new wikiCode(switchboard);

-        serverObjects prop = new serverObjects();

-        if (post == null || !post.containsKey("html"))

-            prop.setLocalized(false);

-        prop.put("rejected", "0");

-        //int showRejectedCount = 10;

-       

-        yacySeed initiator;

-       

-        //indexing queue
-        prop.putNum("indexingSize", sb.getThread(plasmaSwitchboard.INDEXER).getJobCount() + sb.webIndex.queuePreStack.getActiveQueueSize());

-        prop.putNum("indexingMax", (int) sb.getConfigLong(plasmaSwitchboard.INDEXER_SLOTS, 30));

-        prop.putNum("urlpublictextSize", sb.webIndex.countURL());

-        prop.putNum("rwipublictextSize", sb.webIndex.size());

-        if ((sb.webIndex.queuePreStack.size() == 0) && (sb.webIndex.queuePreStack.getActiveQueueSize() == 0)) {
-            prop.put("list", "0"); //is empty

-        } else {

-            IndexingStack.QueueEntry pcentry;

-            long totalSize = 0;

-            int i=0; //counter

-           

-            // getting all entries that are currently in process

-            ArrayList<IndexingStack.QueueEntry> entryList = new ArrayList<IndexingStack.QueueEntry>();

-            entryList.addAll(sb.webIndex.queuePreStack.getActiveQueueEntries());

-            int inProcessCount = entryList.size();

-           

-            // getting all enqueued entries

-            if ((sb.webIndex.queuePreStack.size() > 0)) {

-                Iterator<IndexingStack.QueueEntry> i1 = sb.webIndex.queuePreStack.entryIterator(false);

-                while (i1.hasNext()) entryList.add(i1.next());

-            }

-           

-            int size = (post == null) ? entryList.size() : post.getInt("num", entryList.size());

-            if (size > entryList.size()) size = entryList.size();

-           

-            int ok = 0;

-            for (i = 0; i < size; i++) {

-                boolean inProcess = i < inProcessCount;

-                pcentry = entryList.get(i);

-                if ((pcentry != null) && (pcentry.url() != null)) {

-                    long entrySize = pcentry.size();

-                    totalSize += entrySize;

-                    initiator = sb.webIndex.seedDB.getConnected(pcentry.initiator());

-                    prop.put("list-indexing_"+i+"_profile", (pcentry.profile() != null) ? pcentry.profile().name() : "deleted");

-                    prop.put("list-indexing_"+i+"_initiator", ((initiator == null) ? "proxy" : initiator.getName()));

-                    prop.put("list-indexing_"+i+"_depth", pcentry.depth());

-                    prop.put("list-indexing_"+i+"_modified", pcentry.getModificationDate());

-                    prop.putHTML("list-indexing_"+i+"_anchor", (pcentry.anchorName()==null) ? "" : pcentry.anchorName(), true);

-                    prop.putHTML("list-indexing_"+i+"_url", pcentry.url().toNormalform(false, true), true);

-                    prop.putNum("list-indexing_"+i+"_size", entrySize);

-                    prop.put("list-indexing_"+i+"_inProcess", (inProcess) ? "1" : "0");

-                    prop.put("list-indexing_"+i+"_hash", pcentry.urlHash());

-                    ok++;

-                }

-            }

-            prop.put("list-indexing", ok);

-        }

-       

-        //loader queue
-        prop.put("loaderSize", Integer.toString(sb.crawlQueues.size()));       

-        prop.put("loaderMax", sb.getConfig(plasmaSwitchboard.CRAWLER_THREADS_ACTIVE_MAX, "10"));
-        if (sb.crawlQueues.size() == 0) {

-            prop.put("list-loader", "0");

-        } else {

-            CrawlEntry[] w = sb.crawlQueues.activeWorkerEntries();

-            int count = 0;

-            for (int i = 0; i < w.length; i++)  {

-                if (w[i] == null) continue;

-                prop.put("list-loader_"+count+"_profile", w[i].profileHandle());

-                initiator = sb.webIndex.seedDB.getConnected(w[i].initiator());

-                prop.put("list-loader_"+count+"_initiator", ((initiator == null) ? "proxy" : initiator.getName()));

-                prop.put("list-loader_"+count+"_depth", w[i].depth());

-                prop.putHTML("list-loader_"+count+"_url", w[i].url().toString(), true);

-                count++;

-            }

-            prop.put("list-loader", count);

-        }

-       

-        //local crawl queue
-        prop.putNum("localCrawlSize", Integer.toString(sb.getThread(plasmaSwitchboard.CRAWLJOB_LOCAL_CRAWL).getJobCount()));

-        prop.put("localCrawlState", sb.crawlJobIsPaused(plasmaSwitchboard.CRAWLJOB_LOCAL_CRAWL) ? STATE_PAUSED : STATE_RUNNING);

-        int stackSize = sb.crawlQueues.noticeURL.stackSize(NoticedURL.STACK_TYPE_CORE);

-        addNTable(sb, prop, "list-local", sb.crawlQueues.noticeURL.top(NoticedURL.STACK_TYPE_CORE, Math.min(10, stackSize)));

-

-        //global crawl queue
-        prop.putNum("limitCrawlSize", Integer.toString(sb.crawlQueues.limitCrawlJobSize()));

-        prop.put("limitCrawlState", STATE_RUNNING);

-        stackSize = sb.crawlQueues.noticeURL.stackSize(NoticedURL.STACK_TYPE_LIMIT);

-

-        //global crawl queue

-        prop.putNum("remoteCrawlSize", Integer.toString(sb.getThread(plasmaSwitchboard.CRAWLJOB_REMOTE_TRIGGERED_CRAWL).getJobCount()));

-        prop.put("remoteCrawlState", sb.crawlJobIsPaused(plasmaSwitchboard.CRAWLJOB_REMOTE_TRIGGERED_CRAWL) ? STATE_PAUSED : STATE_RUNNING);

-        stackSize = sb.crawlQueues.noticeURL.stackSize(NoticedURL.STACK_TYPE_LIMIT);

-
-        if (stackSize == 0) {

-            prop.put("list-remote", "0");

-        } else {

-            addNTable(sb, prop, "list-remote", sb.crawlQueues.noticeURL.top(NoticedURL.STACK_TYPE_LIMIT, Math.min(10, stackSize)));

-        }

-

-        // return rewrite properties

-        return prop;

-    }

-   

-   

-    public static final void addNTable(plasmaSwitchboard sb, serverObjects prop, String tableName, CrawlEntry[] crawlerList) {

-

-        int showNum = 0;

-        CrawlEntry urle;

-        yacySeed initiator;

-        for (int i = 0; i < crawlerList.length; i++) {

-            urle = crawlerList[i];

-            if ((urle != null) && (urle.url() != null)) {

-                initiator = sb.webIndex.seedDB.getConnected(urle.initiator());

-                prop.put(tableName + "_" + showNum + "_profile", urle.profileHandle());

-                prop.put(tableName + "_" + showNum + "_initiator", ((initiator == null) ? "proxy" : initiator.getName()));

-                prop.put(tableName + "_" + showNum + "_depth", urle.depth());

-                prop.put(tableName + "_" + showNum + "_modified", daydate(urle.loaddate()));

-                prop.putHTML(tableName + "_" + showNum + "_anchor", urle.name(), true);

-                prop.putHTML(tableName + "_" + showNum + "_url", urle.url().toNormalform(false, true), true);

-                prop.put(tableName + "_" + showNum + "_hash", urle.url().hash());

-                showNum++;

-            }

-        }

-        prop.put(tableName, showNum);

-

-    }

-}

Index: htroot/xml/crawler_p.xml
===================================================================
--- htroot/xml/crawler_p.xml   (Revision 4957)
+++ htroot/xml/crawler_p.xml   (Arbeitskopie)
@@ -1,84 +1,96 @@
<?xml version="1.0"?>
-<queues>
-<dbsize>
-  <urlpublictext>#[urlpublictextSize]#</urlpublictext>
-  <rwipublictext>#[rwipublictextSize]#</rwipublictext>
-</dbsize>
-<indexingqueue>
-  <size>#[indexingSize]#</size>
-  <max>#[indexingMax]#</max>
-#{list-indexing}#
-  <entry>
-    <profile>#[profile]#</profile>
-    <initiator>#[initiator]#</initiator>
-    <depth>#[depth]#</depth>
-    <modified>#[modified]#</modified>
-    <anchor>#[anchor]#</anchor>
-    <url>#[url]#</url>
-    <size>#[size]#</size>
-    <hash>#[hash]#</hash>
-    <inProcess>#(inProcess)#false::true#(/inProcess)#</inProcess>
-  </entry>
-#{/list-indexing}#
-</indexingqueue>
-<loaderqueue>
-  <size>#[loaderSize]#</size>
-  <max>#[loaderMax]#</max>
-#{list-loader}#
-  <entry>
-    <profile>#[profile]#</profile>
-    <initiator>#[initiator]#</initiator>
-    <depth>#[depth]#</depth>
-    <url>#[url]#</url>
-  </entry>
-#{/list-loader}#
-</loaderqueue>
-<localcrawlerqueue>
-  <size>#[localCrawlSize]#</size>
-  <state>#[localCrawlState]#</state>
-#{list-local}#
-  <entry>
-    <profile>#[profile]#</profile>
-    <initiator>#[initiator]#</initiator>
-    <depth>#[depth]#</depth>
-    <modified>#[modified]#</modified>
-    <anchor>#[anchor]#</anchor>
-    <url>#[url]#</url>
-    <hash>#[hash]#</hash>
-    <inProcess>#(inProcess)#false::true#(/inProcess)#</inProcess>
-  </entry>
-#{/list-local}#
-</localcrawlerqueue>
-<limitcrawlerqueue>
-  <size>#[limitCrawlSize]#</size>
-  <state>#[limitCrawlState]#</state>
-#{list-limit}#
-  <entry>
-    <profile>#[profile]#</profile>
-   <initiator>#[initiator]#</initiator>
-    <depth>#[depth]#</depth>
-    <modified>#[modified]#</modified>
-    <anchor>#[anchor]#</anchor>
-    <url>#[url]#</url>
-    <hash>#[hash]#</hash>
-    <inProcess>#(inProcess)#false::true#(/inProcess)#</inProcess>
-  </entry>
-#{/list-limit}#
-</limitcrawlerqueue>
-<remotecrawlerqueue>
-  <size>#[remoteCrawlSize]#</size>
-  <state>#[remoteCrawlState]#</state>
-#{list-remote}#
-  <entry>
-    <profile>#[profile]#</profile>
-   <initiator>#[initiator]#</initiator>
-    <depth>#[depth]#</depth>
-    <modified>#[modified]#</modified>
-    <anchor>#[anchor]#</anchor>
-    <url>#[url]#</url>
-    <hash>#[hash]#</hash>
-    <inProcess>#(inProcess)#false::true#(/inProcess)#</inProcess>
-  </entry>
-#{/list-remote}#
-</remotecrawlerqueue>
-</queues>
+<crawler>
+  <status>
+    <ppm>#[ppm]#</ppm>
+    <wordCacheWSize>#[wordCacheWSize]#</wordCacheWSize>
+    <wordCacheWCount>#[wordCacheWCount]#</wordCacheWCount>
+    <wordCacheMaxSize>#[wordCacheMaxSize]#</wordCacheMaxSize>
+    <wordCacheMaxCount>#[wordCacheMaxCount]#</wordCacheMaxCount>
+    <traffic>
+     <crawler>#[trafficCrawler]#</crawler>
+    </traffic>
+  </status>
+  <queues>
+    <dbsize>
+      <urlpublictext>#[urlpublictextSize]#</urlpublictext>
+      <rwipublictext>#[rwipublictextSize]#</rwipublictext>
+    </dbsize>
+    <indexingqueue>
+      <size>#[indexingSize]#</size>
+      <max>#[indexingMax]#</max>
+      #{list-indexing}#
+      <entry>
+        <profile>#[profile]#</profile>
+        <initiator>#[initiator]#</initiator>
+        <depth>#[depth]#</depth>
+        <modified>#[modified]#</modified>
+        <anchor>#[anchor]#</anchor>
+        <url>#[url]#</url>
+        <size>#[size]#</size>
+        <hash>#[hash]#</hash>
+        <inProcess>#(inProcess)#false::true#(/inProcess)#</inProcess>
+      </entry>
+      #{/list-indexing}#
+    </indexingqueue>
+    <loaderqueue>
+      <size>#[loaderSize]#</size>
+      <max>#[loaderMax]#</max>
+      #{list-loader}#
+      <entry>
+        <profile>#[profile]#</profile>
+        <initiator>#[initiator]#</initiator>
+        <depth>#[depth]#</depth>
+        <url>#[url]#</url>
+      </entry>
+      #{/list-loader}#
+    </loaderqueue>
+    <localcrawlerqueue>
+      <size>#[localCrawlSize]#</size>
+      <state>#[localCrawlState]#</state>
+      #{list-local}#
+      <entry>
+        <profile>#[profile]#</profile>
+        <initiator>#[initiator]#</initiator>
+        <depth>#[depth]#</depth>
+        <modified>#[modified]#</modified>
+        <anchor>#[anchor]#</anchor>
+        <url>#[url]#</url>
+        <hash>#[hash]#</hash>
+        <inProcess>#(inProcess)#false::true#(/inProcess)#</inProcess>
+      </entry>
+      #{/list-local}#
+    </localcrawlerqueue>
+    <limitcrawlerqueue>
+      <size>#[limitCrawlSize]#</size>
+      <state>#[limitCrawlState]#</state>
+      #{list-limit}#
+      <entry>
+        <profile>#[profile]#</profile>
+        <initiator>#[initiator]#</initiator>
+        <depth>#[depth]#</depth>
+        <modified>#[modified]#</modified>
+        <anchor>#[anchor]#</anchor>
+        <url>#[url]#</url>
+        <hash>#[hash]#</hash>
+        <inProcess>#(inProcess)#false::true#(/inProcess)#</inProcess>
+      </entry>
+      #{/list-limit}#
+    </limitcrawlerqueue>
+    <remotecrawlerqueue>
+      <size>#[remoteCrawlSize]#</size>
+      <state>#[remoteCrawlState]#</state>
+      #{list-remote}#
+      <entry>
+        <profile>#[profile]#</profile>
+        <initiator>#[initiator]#</initiator>
+        <depth>#[depth]#</depth>
+        <modified>#[modified]#</modified>
+        <anchor>#[anchor]#</anchor>
+        <url>#[url]#</url>
+        <hash>#[hash]#</hash>
+        <inProcess>#(inProcess)#false::true#(/inProcess)#</inProcess>
+      </entry>
+      #{/list-remote}#
+    </remotecrawlerqueue>
+  </queues>
+</crawler>
datengrab
 
Beiträge: 366
Registriert: Sa Aug 25, 2007 7:22 pm

Re: Status autoupdate

Beitragvon datengrab » So Jun 29, 2008 2:06 pm

???
datengrab
 
Beiträge: 366
Registriert: Sa Aug 25, 2007 7:22 pm


Zurück zu Mitmachen

Wer ist online?

Mitglieder in diesem Forum: 0 Mitglieder und 2 Gäste