Speicherverbrauch dauerthaft senken ...

Keine Scheu, hier darf alles gefragt und diskutiert werden. Das ist das Forum für YaCy-Anfänger. Hier kann man 'wo muss man klicken' fragen und sich über Grundlagen zur Suchmaschinentechnik unterhalten.
Forumsregeln
Hier werden Fragen beantwortet und wir versuchen die Probleme von YaCy-Newbies zu klären. Bitte beantwortete Fragen im YaCy-Wiki http://wiki.yacy.de dokumentieren!

Speicherverbrauch dauerthaft senken ...

Beitragvon Frabi » Di Jul 24, 2012 8:19 am

Hallo,

an welchen Schräubchen kann man drehen um zu den Speicherverbrauch dauerhaft zu senken.

Mein Peer kam anfangs mit 500 MB gut aus, mittlerweile laufe ich aber auch mit 800MB dauerhaft in die Abschaltung des DHT-Empfangs.

Kann man da was dran ändern?
Frabi
 
Beiträge: 7
Registriert: So Dez 04, 2011 2:13 pm

Re: Speicherverbrauch dauerthaft senken ...

Beitragvon gaston » Di Jul 24, 2012 10:14 am

Auf der Seite PerformanceMemory_p.html kannst du sehen wie viel Speicher YaCy für den Index braucht, der immer komplett im Speicher landet (oder nicht?).

Mein Peer 28112011 verbraucht aktuell 700,12 MB und 1.303,88 MB, also unter 2.5GB Speicher würde bei mir YaCy wahrscheinlich nicht mehr starten.

Was bei mir eine Zeit lang geholfen hat, steht in mein YaCy Profile.

Um Speicher sparen zu können muss YaCy wahrscheinlich umgebaut werden, leider. Habe bei mir z.B. die Resourcenüberwachung und das TrayIcon entfernt, spart auch etwas Speicher ;)
gaston
 
Beiträge: 143
Registriert: Fr Jan 06, 2012 2:22 pm

Re: Speicherverbrauch dauerthaft senken ...

Beitragvon Orbiter » Di Jul 24, 2012 11:01 am

ich hab in den letzten Wochen viel mit dem MemoryAnalyzer ( http://www.eclipse.org/mat/ ) gearbeitet und dabei einiges an potentiellen und tatsächlichen memory leaks gefunden und behoben. Wenn ihr Lust habt könnt ihr den Dingen ja damit mal versuchen auf den Grund zu gehen.
Dazu ist aber ein wenig Code-Affiniät sinnvoll.
Orbiter
 
Beiträge: 5792
Registriert: Di Jun 26, 2007 10:58 pm
Wohnort: Frankfurt am Main

Re: Speicherverbrauch dauerthaft senken ...

Beitragvon gaston » Di Jul 24, 2012 11:28 am

Ich benutze immer "Java VisualVM", was mir dabei auffällt ist das man kaum Speicher sparen kann ohne den Index nicht komplett im Speicher zu halten. YaCy verbraucht sehr schnell, sehr viel"temporären" Speicher durch einfache int Objekte. Ich habe deswegen bei mir Base64Order geändert, was ja ständig benutzt wird. Durch die dadurch nötigen synchronized's dauert der Start zwar etwas länger, aber ansonsten merkt man davon nichts, außer das weniger Speicher verbrauch wird ;)

Wenn es einer haben möchte...
gaston
 
Beiträge: 143
Registriert: Fr Jan 06, 2012 2:22 pm

Re: Speicherverbrauch dauerthaft senken ...

Beitragvon Frabi » Di Jul 24, 2012 11:58 am

gaston hat geschrieben:Auf der Seite PerformanceMemory_p.html kannst du sehen wie viel Speicher YaCy für den Index braucht, der immer komplett im Speicher landet (oder nicht?).

Mein Peer 28112011 verbraucht aktuell 700,12 MB und 1.303,88 MB, also unter 2.5GB Speicher würde bei mir YaCy wahrscheinlich nicht mehr starten.


Ich verbrauche für meinen Index ungefähr 500MB, ich habe jetzt erstmal das Crawlen ausgestellt. Mein Word-Index fällt dadurch stündlich um ca. 1.100 Wörter. Ob das alleine auf Dauer hilft werde ich sehen. Mehr Speicher kann ich nicht zuweisen, da mein Server nur 1GB hat.

Da ein Neustart nicht wirklich eine Verbesserung bringt glaube ich nicht das es an temporären Objekten liegt.
Frabi
 
Beiträge: 7
Registriert: So Dez 04, 2011 2:13 pm

Re: Speicherverbrauch dauerthaft senken ...

Beitragvon Orbiter » Di Jul 24, 2012 1:45 pm

gaston hat geschrieben:YaCy verbraucht sehr schnell, sehr viel"temporären" Speicher durch einfache int Objekte.

wenn du mit 'temporären Speicher' einfach nur kurzlebige Objekte ohne weitere Bindung nach einem Methodenaufruf meinst ist das ja wurscht. Das pusht zwar die Speicherbelegung im Heap hoch, wird aber alles wirder beim nächsten GC freigegeben.

gaston hat geschrieben:Ich habe deswegen bei mir Base64Order geändert, was ja ständig benutzt wird. Durch die dadurch nötigen synchronized's dauert der Start zwar etwas länger, aber ansonsten merkt man davon nichts, außer das weniger Speicher verbrauch wird ;)

Zeig mal.
Orbiter
 
Beiträge: 5792
Registriert: Di Jun 26, 2007 10:58 pm
Wohnort: Frankfurt am Main

Re: Speicherverbrauch dauerthaft senken ...

Beitragvon gaston » Di Jul 24, 2012 1:57 pm

Orbiter hat geschrieben:
gaston hat geschrieben:YaCy verbraucht sehr schnell, sehr viel"temporären" Speicher durch einfache int Objekte.

wenn du mit 'temporären Speicher' einfach nur kurzlebige Objekte ohne weitere Bindung nach einem Methodenaufruf meinst ist das ja wurscht. Das pusht zwar die Speicherbelegung im Heap hoch, wird aber alles wirder beim nächsten GC freigegeben.

Genau das, aber das ganze freigeben/verwalten kostet aber auch Zeit ;)

Die Anordnung der Variablen ...sollen Probleme beim updaten verhindern. Git verschluckt sich bei mir des öfteren .
Code: Alles auswählen
// Base64Order.java
// -----------------------
// part of The Kelondro Database
// (C) by Michael Peter Christen; mc@yacy.net
// first published on http://www.anomic.de
// Frankfurt, Germany, 2005
// created 03.01.2006
//
// $LastChangedDate$
// $LastChangedRevision$
// $LastChangedBy$
//
// This program is free software; you can redistribute it and/or modify
// it under the terms of the GNU General Public License as published by
// the Free Software Foundation; either version 2 of the License, or
// (at your option) any later version.
//
// This program is distributed in the hope that it will be useful,
// but WITHOUT ANY WARRANTY; without even the implied warranty of
// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
// GNU General Public License for more details.
//
// You should have received a copy of the GNU General Public License
// along with this program; if not, write to the Free Software
// Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA  02111-1307  USA


package net.yacy.kelondro.order;

import java.io.Serializable;
import java.util.Comparator;

import net.yacy.cora.document.UTF8;
import net.yacy.cora.order.AbstractOrder;
import net.yacy.cora.order.ByteOrder;
import net.yacy.cora.order.Order;


public class Base64Order extends AbstractOrder<byte[]> implements ByteOrder, Comparator<byte[]>, Cloneable, Serializable {

    private static final long serialVersionUID=980647587445343851L;

    public  static final byte[] alpha_standard = "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/".getBytes();
    public  static final byte[] alpha_enhanced = "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789-_".getBytes();
    private static final byte[] ahpla_standard = new byte[128];
    private static final byte[] ahpla_enhanced = new byte[128];

    static {
        for (int i = 0; i < 128; i++) {
            Base64Order.ahpla_standard[i] = -1;
            Base64Order.ahpla_enhanced[i] = -1;
        }
        for (int i = 0; i < Base64Order.alpha_standard.length; i++) {
            Base64Order.ahpla_standard[Base64Order.alpha_standard[i]] = (byte) i;
            Base64Order.ahpla_enhanced[Base64Order.alpha_enhanced[i]] = (byte) i;
        }
    }

    public static final Base64Order standardCoder = new Base64Order(true, true);
    public static final Base64Order enhancedCoder = new Base64Order(true, false);

    private final boolean rfc1113;
    private final byte[] alpha;
    private final byte[] ahpla;
    private final byte[] ab; // decision table for comparisments

    public Base64Order(final boolean up, final boolean rfc1113compliant) {
        // if we choose not to be rfc1113compliant,
        // then we get shorter base64 results which are also filename-compatible
        this.rfc1113 = rfc1113compliant;
        this.asc = up;
        this.alpha = rfc1113compliant ? Base64Order.alpha_standard : Base64Order.alpha_enhanced;
        this.ahpla = rfc1113compliant ? Base64Order.ahpla_standard : Base64Order.ahpla_enhanced;
        this.ab = new byte[1 << 14];
        byte acc, bcc;
        byte c;
        // pre-compute comparisment results: this omits one single ahpla lookup during comparisment
        for (final byte ac : this.alpha) {
            for (final byte bc : this.alpha) {
                acc = this.ahpla[ac];
                bcc = this.ahpla[bc];
                c = 0;
                if (acc > bcc) { c = 1; }
                if (acc < bcc) { c = -1; }
                this.ab[(ac << 7) | bc] = c;
            }
        }
    }

    public static byte[] zero(int length) {
        int len = length;
        final byte[] z = new byte[len];
        while (len > 0) {
            len--; z[len] = Base64Order.alpha_standard[0];
        }
        return z;
    }

    @Override
    public final Order<byte[]> clone() {
        final Base64Order o = new Base64Order(this.asc, this.rfc1113);
        o.rotate(this.zero);
        return o;
    }

    @Override
    public final boolean wellformed(final byte[] a) {
        return this.wellformed(a, 0, a.length);
    }

    @Override
    public final synchronized boolean wellformed(final byte[] a, final int astart, final int alength) {
        assert astart + alength <= a.length : "astart = " + astart + ", alength = " + alength + ", a.length = " + a.length;
        int b;
        for (int i = astart + alength - 1; i >= astart; i--) {
            b = a[i];
            if ((b < 0) || (b >= 128) || (this.ahpla[b] == -1)) {
                return false;
            }
        }
        return true;
    }

    public static final ByteOrder bySignature(final String signature) {
        if ("Bd".equals(signature)) return new Base64Order(false, false);
        if ("bd".equals(signature)) return new Base64Order(false, true);
        if ("Bu".equals(signature)) return new Base64Order(true, false);
        if ("bu".equals(signature)) return new Base64Order(true, true);
        return null;
    }

    @Override
    public final String signature() {
        if ((!this.asc) && (!this.rfc1113)) return "Bd";
        if ((!this.asc) && ( this.rfc1113)) return "bd";
        if (( this.asc) && (!this.rfc1113)) return "Bu";
        if (( this.asc) && ( this.rfc1113)) return "bu";
        return null;
    }

    public final char encodeByte(final byte b) {
        return (char) this.alpha[b];
    }

    public final byte decodeByte(final byte b) {
        return this.ahpla[b];
    }

    public final byte decodeByte(final char b) {
        return this.ahpla[b];
    }

    public final StringBuilder encodeLongSB(final long c, final int length) {
        int  len = length;
        long enc = c;
        final StringBuilder s = new StringBuilder(len);
        s.setLength(len);
        while (len > 0) {
            s.setCharAt(--len, (char) this.alpha[(byte) (enc & 0x3F)]);
            enc >>= 6;
        }
        return s;
    }

    public final byte[] encodeLongBA(final long c, final int length) {
        int  len = length;
        long enc = c;
        final byte[] s = new byte[len];
        while (len > 0) {
            s[--len] = this.alpha[(byte) (enc & 0x3F)];
            enc >>= 6;
        }
        return s;
    }

    public final byte[] encodeLongSubstr(final long c, final int length) {
        int  len = length;
        long enc = c;
        final byte[] s = new byte[len];
        while (len > 0) {
            s[--len] = this.alpha[(byte) (enc & 0x3F)];
            enc >>= 6;
        }
        return s;
    }

    public final void encodeLong(final long c, final byte[] b, final int offset, final int length) {
        assert offset + length <= b.length;
        int  len = length;
        long enc = c;
        while (len > 0) {
            b[--len + offset] = this.alpha[(byte) (enc & 0x3F)];
            enc >>= 6;
        }
    }

    private static int d0_l, d0_i;
    private static long d0_c;
    public final synchronized long decodeLong(String s) {
        Base64Order.d0_l = s.length();
        while (Base64Order.d0_l-- > 0 && s.charAt(Base64Order.d0_l) == '=') { continue; }
        Base64Order.d0_l += 1;
        Base64Order.d0_c = 0;
        for (Base64Order.d0_i = 0; Base64Order.d0_i < Base64Order.d0_l; Base64Order.d0_i++) {
            Base64Order.d0_c = (Base64Order.d0_c << 6) | this.ahpla[s.charAt(Base64Order.d0_i)];
        }
        return Base64Order.d0_c;
    }

    private static int d1_i;
    private static long d1_c;
    public final synchronized long decodeLong(final byte[] s, final int offset, final int length) {
        int  len = length;
        while ((len > 0) && (s[offset + length - 1] == '=')) { len--; }
        d1_c = 0;
        for (d1_i = 0; d1_i < len; d1_i++) {
            d1_c = (d1_c << 6) | this.ahpla[s[offset + d1_i]];
        }
        return d1_c;
    }

    private static int m_i;
    private static long m_c;
    public static synchronized long max(final int len) {
        // computes the maximum number that can be coded with a base64-encoded
        // String of base len
        m_c = 0;
        for (m_i = 0; m_i < len; m_i++) {
            m_c = (m_c << 6) | 63;
        }
        return m_c;
    }

    public final String encodeString(final String in) {
        return encode(UTF8.getBytes(in));
    }

    // we will use this encoding to encode strings with 2^8 values to
    // b64-Strings
    // we will do that by grouping each three input bytes to four output bytes.
    private static int e0_pos;
    private static long e0_l;
    private static StringBuilder e0_out = new StringBuilder(1024);
    public final synchronized String encode(final byte[] in) {
        e0_out.setLength(0);
        if (in == null || in.length == 0) {
            return e0_out.toString();
        }
//        e0_lene = in.length / 3 * 4 + 3;
//        StringBuilder out = new StringBuilder(e0_lene);
        e0_pos = 0;
        while (in.length - e0_pos >= 3) {
            e0_l = ((((0XffL & in[e0_pos]) << 8) + (0XffL & in[e0_pos + 1])) << 8) + (0XffL & in[e0_pos + 2]);
            e0_pos += 3;
            e0_out = e0_out.append(encodeLongSB(e0_l, 4));
        }
        // now there may be remaining bytes
        if (in.length % 3 != 0) {
            e0_out = e0_out.append((in.length % 3 == 2) ? encodeLongSB((((0XffL & in[e0_pos]) << 8) + (0XffL & in[e0_pos + 1])) << 8, 4).substring(0, 3) : encodeLongSB((((0XffL & in[e0_pos])) << 8) << 8, 4).substring(0, 2));
        }
        if (this.rfc1113) {
            while (e0_out.length() % 4 > 0) {
                e0_out.append("=");
            }
        }
        // return result
        //assert lene == out.length() : "lene = " + lene + ", out.len = " + out.length();
//        System.out.println("DEBUG: BASE64 encode length=" + e0_out.length());
        return e0_out.toString();
    }

    private static int e1Writepos, e1Pos;
    private static long e1L;
    public final synchronized byte[] encodeSubstring(final byte[] in, final int sublen) {
        if (in.length == 0) {
            return null;
        }
        final byte[] out = new byte[sublen];
        Base64Order.e1Writepos = 0;
        Base64Order.e1Pos = 0;
        while (in.length - Base64Order.e1Pos >= 3 && Base64Order.e1Writepos < sublen) {
            Base64Order.e1L = ((((0XffL & in[Base64Order.e1Pos]) << 8) + (0XffL & in[Base64Order.e1Pos + 1])) << 8) + (0XffL & in[Base64Order.e1Pos + 2]);
            Base64Order.e1Pos += 3;
            System.arraycopy(this.encodeLongSubstr(Base64Order.e1L, 4), 0, out, Base64Order.e1Writepos, 4);
            Base64Order.e1Writepos += 4;
        }
        // now there may be remaining bytes
        if (in.length % 3 != 0 && Base64Order.e1Writepos < sublen) {
            if (in.length % 3 == 2) {
                System.arraycopy(encodeLongBA((((0XffL & in[Base64Order.e1Pos]) << 8) + (0XffL & in[Base64Order.e1Pos + 1])) << 8, 4), 0, out, Base64Order.e1Writepos, 3);
                Base64Order.e1Writepos += 3;
            } else {
                System.arraycopy(encodeLongBA((((0XffL & in[Base64Order.e1Pos])) << 8) << 8, 4), 0, out, Base64Order.e1Writepos, 2);
                Base64Order.e1Writepos += 2;
            }
        }

        if (this.rfc1113) {
            while (Base64Order.e1Writepos % 4 > 0 && Base64Order.e1Writepos < sublen) {
                out[Base64Order.e1Writepos] = '=';
            }
        }
        //assert encode(in).substring(0, sublen).equals(ASCII.String(out));
        return out;
    }

    public final String decodeString(final String in) {
        return UTF8.String(decode(in));
    }

    private static int d1PosIn, d1PosOut;
    private static long d1L;
    public final synchronized byte[] decode(String in) {
        if ((in == null) || (in.length() == 0)) return new byte[0];
        try {
            Base64Order.d1PosIn = 0;
            Base64Order.d1PosOut = 0;
            if (this.rfc1113) while (in.charAt(in.length() - 1) == '=') in = in.substring(0, in.length() - 1);
            final byte[] out = new byte[in.length() / 4 * 3 + (((in.length() % 4) == 0) ? 0 : in.length() % 4 - 1)];
            while (d1PosIn + 3 < in.length()) {
                d1L = decodeLong(in.substring(d1PosIn, d1PosIn + 4));
                out[d1PosOut + 2] = (byte) (d1L % 256);
                d1L = d1L / 256;
                out[d1PosOut + 1] = (byte) (d1L % 256);
                d1L = d1L / 256;
                out[d1PosOut] = (byte) (d1L % 256);
                d1L = d1L / 256;
                d1PosIn += 4;
                d1PosOut += 3;
            }
            if (d1PosIn < in.length()) {
                if (in.length() - d1PosIn == 3) {
                    d1L = decodeLong(in.substring(d1PosIn) + "A");
                    d1L = d1L / 256;
                    out[d1PosOut + 1] = (byte) (d1L % 256);
                    d1L = d1L / 256;
                    out[d1PosOut] = (byte) (d1L % 256);
                    d1L = d1L / 256;
                } else {
                    d1L = decodeLong(in.substring(d1PosIn) + "AA");
                    d1L = d1L / 256 / 256;
                    out[d1PosOut] = (byte) (d1L % 256);
                    d1L = d1L / 256;
                }
            }
            return out;
        } catch (final ArrayIndexOutOfBoundsException e) {
            // maybe the input was not base64
            // TODO: Throw exception again
            // throw new RuntimeException("input probably not base64");
            System.err.println("wrong string receive: " + in);
            return new byte[0];
        }
    }

    private static long c0C;
    private static int c0P;
    private static byte c0B;
    private synchronized long cardinalI(final String key) {
        // returns a cardinal number in the range of 0 .. Long.MAX_VALUE
        Base64Order.c0C = 0;
        Base64Order.c0P = 0;
        while ((Base64Order.c0P < 10) && (Base64Order.c0P < key.length())) {
            Base64Order.c0B = this.ahpla[key.charAt(Base64Order.c0P++)];
            if (Base64Order.c0B < 0) {
                return -1;
            }
            Base64Order.c0C = (Base64Order.c0C << 6) | Base64Order.c0B;
        }
        while (Base64Order.c0P++ < 10) {
            Base64Order.c0C = Base64Order.c0C << 6;
        }
        Base64Order.c0C = (Base64Order.c0C << 3) | 7;
        assert Base64Order.c0C >= 0;
        return Base64Order.c0C;
    }

    private static long c1C;
    private static int c1Lim, c1Lim10;
    private static byte c1B;
    private synchronized long cardinalI(final byte[] key, final int offset, final int len) {
        // returns a cardinal number in the range of 0 .. Long.MAX_VALUE
        int off = offset;
        Base64Order.c1C = 0;
        Base64Order.c1Lim = off + Math.min(10, len);
        Base64Order.c1Lim10 = off + 10;
        while (off < Base64Order.c1Lim) {
            Base64Order.c1B = key[off++];
            if (Base64Order.c1B < 0) {
                return -1;
            }
            Base64Order.c1B = this.ahpla[Base64Order.c1B];
            if (Base64Order.c1B < 0) {
                return -1;
            }
            Base64Order.c1C = (Base64Order.c1C << 6) | Base64Order.c1B;
        }
        while (off++ < Base64Order.c1Lim10) {
            Base64Order.c1C = Base64Order.c1C << 6;
        }
        Base64Order.c1C = (Base64Order.c1C << 3) | 7;
        assert Base64Order.c1C >= 0;
        return Base64Order.c1C;
    }

    public final byte[] uncardinal(long number) {
        long c = number;
        c = c >> 3;
        final byte[] b = new byte[12];
        for (int p = 9; p >= 0; p--) {
            b[p] = this.alpha[(int) (c & 0x3fL)];
            c = c >> 6;
        }
        b[10] = this.alpha[0x3f];
        b[11] = this.alpha[0x3f];
        return b;
    }

    @Override
    public final long cardinal(final byte[] key) {
        if (this.zero == null) return cardinalI(key, 0, key.length);
        final long zeroCardinal = cardinalI(this.zero, 0, this.zero.length);
        final long keyCardinal = cardinalI(key, 0, key.length);
        if (keyCardinal > zeroCardinal) return keyCardinal - zeroCardinal;
        return Long.MAX_VALUE - keyCardinal + zeroCardinal;
    }

    @Override
    public final long cardinal(final byte[] key, final int off, final int len) {
        if (this.zero == null) return cardinalI(key, off, len);
        final long zeroCardinal = cardinalI(this.zero, 0, this.zero.length);
        final long keyCardinal = cardinalI(key, off, len);
        if (keyCardinal > zeroCardinal) return keyCardinal - zeroCardinal;
        return Long.MAX_VALUE - keyCardinal + zeroCardinal;
    }

    public final long cardinal(final String key) {
        if (this.zero == null) return cardinalI(key);
        final long zeroCardinal = cardinalI(this.zero, 0, this.zero.length);
        final long keyCardinal = cardinalI(key);
        if (keyCardinal > zeroCardinal) return keyCardinal - zeroCardinal;
        return Long.MAX_VALUE - keyCardinal + zeroCardinal;
    }

    private static int sig(final int x) {
        return (x > 0) ? 1 : (x < 0) ? -1 : 0;
    }

    private static int e0_astart, e0_bstart, e0_length;
    @Override
    public synchronized final boolean equal(final byte[] a, final byte[] b) {
        if ((a == null) && (b == null)) return true;
        if ((a == null) || (b == null)) return false;
        if (a.length != b.length) return false;
        e0_astart = 0;
        e0_bstart = 0;
        e0_length = a.length;
        while (e0_length-- != 0) {
            if (a[e0_astart++] != b[e0_bstart++]) return false;
        }
        return true;
    }

    @Override
    public final boolean equal(final byte[] a, int astart, final byte[] b, int bstart, final int length) {
        if ((a == null) && (b == null)) return true;
        if ((a == null) || (b == null)) return false;
        int len = length;
        while (len-- != 0) {
            if (a[astart++] != b[bstart++]) return false;
        }
        return true;
    }

    @Override
    public final int compare(final byte[] a, final byte[] b) {
        try {
        return (this.asc) ?
                ((this.zero == null) ? compares(a, b) : compare0(a, b, a.length))
                :
                ((this.zero == null) ? compares(b, a) : compare0(b, a, a.length));
        } catch (final Throwable e) {
            // if a or b is not well-formed, an ArrayIndexOutOfBoundsException may occur
            // in that case we don't want that the exception makes databse functions
            // unusable and effective creates a showstopper. In such cases we apply
            // a different order on the objects and treat not well-formed objects
            // as bigger as all others. If both object are not well-formed, they are
            // compared with the natural order.
            final boolean wfa = wellformed(a);
            final boolean wfb = wellformed(b);
            if (wfa && wfb) {
                // uh strange. throw the exception
                if (e instanceof ArrayIndexOutOfBoundsException) throw (ArrayIndexOutOfBoundsException) e;
                throw new RuntimeException(e.getMessage());
            }
            if (wfa) return (this.asc) ? -1 :  1;
            if (wfb) return (this.asc) ?  1 : -1;
            return ((this.asc) ? 1 : -1) * NaturalOrder.naturalOrder.compare(a, b);
        }
    }

    @Override
    public final int compare(final byte[] a, final byte[] b, final int length) {
        try {
            return (this.asc) ?
                    compare0(a, b, length)
                    :
                    compare0(b, a, length);
        } catch (final Throwable e) {
            // same handling as in simple compare method above
            final boolean wfa = wellformed(a, 0, length);
            final boolean wfb = wellformed(b, 0, length);
            if (wfa && wfb) {
                // uh strange. throw the exception
                if (e instanceof ArrayIndexOutOfBoundsException) throw (ArrayIndexOutOfBoundsException) e;
                throw new RuntimeException(e.getMessage());
            }
            if (wfa) return (this.asc) ? -1 :  1;
            if (wfb) return (this.asc) ?  1 : -1;
            return ((this.asc) ? 1 : -1) * NaturalOrder.naturalOrder.compare(a, b, length);
        }
    }

    @Override
    public final int compare(final byte[] a, final int aoffset, final byte[] b, final int boffset, final int length) {
        try {
            return (this.asc) ?
                    compare0(a, aoffset, b, boffset, length)
                    :
                    compare0(b, boffset, a, aoffset, length);
        } catch (final Throwable e) {
            // same handling as in simple compare method above
            final boolean wfa = wellformed(a, aoffset, length);
            final boolean wfb = wellformed(b, boffset, length);
            if (wfa && wfb) {
                // uh strange. throw the exception
                if (e instanceof ArrayIndexOutOfBoundsException) throw (ArrayIndexOutOfBoundsException) e;
                throw new RuntimeException(e.getMessage());
            }
            if (wfa) return (this.asc) ? -1 :  1;
            if (wfb) return (this.asc) ?  1 : -1;
            return ((this.asc) ? 1 : -1) * NaturalOrder.naturalOrder.compare(a, aoffset, b, boffset, length);
        }
    }

    private int compare0(final byte[] a, final byte[] b, final int length) {
        if (this.zero == null) {
            return compares(a, b, length);
        }
        // we have an artificial start point. check all combinations
        int len = length;
        if (this.zero.length < len) {
            len = this.zero.length;
        }
        final int az = compares(a, this.zero, len); // -1 if a < z; 0 if a == z; 1 if a > z
        final int bz = compares(b, this.zero, len); // -1 if b < z; 0 if b == z; 1 if b > z
        if (az == bz) {
            return compares(a, b, len);
        }
        return sig(az - bz);
    }

    private int compare0(final byte[] a, final int aoffset, final byte[] b, final int boffset, int length) {
        if (this.zero == null) return compares(a, aoffset, b, boffset, length);

        // we have an artificial start point. check all combinations
        if (this.zero.length < length) length = this.zero.length;
        final int az = compares(a, aoffset, this.zero, 0, length); // -1 if a < z; 0 if a == z; 1 if a > z
        final int bz = compares(b, boffset, this.zero, 0, length); // -1 if b < z; 0 if b == z; 1 if b > z
        if (az == bz) return compares(a, aoffset, b, boffset, length);
        return sig(az - bz);
    }

    private static short c2_i, c2_ml;
    private static int c2_al, c2_bl;
    private static byte c2_ac, c2_bc;
    private synchronized final int compares(final byte[] a, final byte[] b) {
        //assert (this.ahpla.length == 128);
        c2_i = 0;
        c2_al = a.length;
        c2_bl = b.length;
        c2_ml = (short) Math.min(c2_al, c2_bl);
        while (c2_i < c2_ml) { // trace point
            //assert (i < a.length) : "i = " + i + ", aoffset = " + 0 + ", a.length = " + a.length + ", a = " + NaturalOrder.arrayList(a, 0, al);
            //assert (i < b.length) : "i = " + i + ", boffset = " + 0 + ", b.length = " + b.length + ", b = " + NaturalOrder.arrayList(b, 0, al);
            c2_ac = a[c2_i];
            //assert (ac >= 0) && (ac < 128) : "ac = " + ac + ", a = " + NaturalOrder.arrayList(a, 0, al);
            c2_bc = b[c2_i];
            //assert (bc >= 0) && (bc < 128) : "bc = " + bc + ", b = " + NaturalOrder.arrayList(b, 0, al);
            //assert ac != 0;
            //assert bc != 0;
            if (c2_ac != c2_bc) return this.ab[(c2_ac << 7) | c2_bc];
            c2_i++;
        }
        // compare length
        if (c2_al > c2_bl) return 1;
        if (c2_al < c2_bl) return -1;
        // they are equal
        return 0;
    }

    private static short c3_i;
    private static byte c3_ac, c3_bc;
    private synchronized final int compares(final byte[] a, final byte[] b, final int length) {
        //assert (length <= a.length) : "a.length = " + a.length + ", alength = " + length;
        //assert (length <= b.length) : "b.length = " + b.length + ", blength = " + length;
        //assert (this.ahpla.length == 128);
        c3_i = 0;
        while (c3_i < length) {
            //assert (i < a.length) : "i = " + i + ", a.length = " + a.length + ", a = " + NaturalOrder.arrayList(a, 0, length);
            //assert (i < b.length) : "i = " + i + ", b.length = " + b.length + ", b = " + NaturalOrder.arrayList(b, 0, length);
            c3_ac = a[c3_i];
            //assert (ac >= 0) && (ac < 128) : "ac = " + ac + ", a = " + NaturalOrder.arrayList(a, 0, length);
            c3_bc = b[c3_i];
            //assert (bc >= 0) && (bc < 128) : "bc = " + bc + ", b = " + NaturalOrder.arrayList(b, 0, length);
            //assert ac != 0;
            //assert bc != 0;
            if (c3_ac != c3_bc) return this.ab[(c3_ac << 7) | c3_bc];
            c3_i++;
        }
        // they are equal
        return 0;
    }

    private static short c4_i;
    private static byte c4_ac, c4_bc;
    private synchronized final int compares(final byte[] a, final int aoffset, final byte[] b, final int boffset, final int length) {
        //assert (aoffset + length <= a.length) : "a.length = " + a.length + ", aoffset = " + aoffset + ", alength = " + length;
        //assert (boffset + length <= b.length) : "b.length = " + b.length + ", boffset = " + boffset + ", blength = " + length;
        //assert (this.ahpla.length == 128);
        c4_i = 0;
        while (c4_i < length) {
            //assert (i + aoffset < a.length) : "i = " + i + ", aoffset = " + aoffset + ", a.length = " + a.length + ", a = " + NaturalOrder.arrayList(a, aoffset, length);
            //assert (i + boffset < b.length) : "i = " + i + ", boffset = " + boffset + ", b.length = " + b.length + ", b = " + NaturalOrder.arrayList(b, boffset, length);
            c4_ac = a[aoffset + c4_i];
            //assert (ac >= 0) && (ac < 128) : "ac = " + ac + ", a = " + NaturalOrder.arrayList(a, aoffset, length);
            c4_bc = b[boffset + c4_i];
            //assert (bc >= 0) && (bc < 128) : "bc = " + bc + ", b = " + NaturalOrder.arrayList(b, boffset, length);
            //assert ac != 0;
            //assert bc != 0;
            if (c4_ac != c4_bc) return this.ab[(c4_ac << 7) | c4_bc];
            c4_i++;
        }
        // they are equal
        return 0;
    }
/*
    public static void main(final String[] s) {
        // java -classpath classes de.anomic.kelondro.kelondroBase64Order
        final Base64Order b64 = new Base64Order(true, true);
        if (s.length == 0) {
            System.out.println("usage: -[ec|dc|es|ds|clcn] <arg>");
            System.exit(0);
        }
        if ("-ec".equals(s[0])) {
            // generate a b64 encoding from a given cardinal
            System.out.println(b64.encodeLongSB(Long.parseLong(s[1]), 4));
        }
        if ("-dc".equals(s[0])) {
            // generate a b64 decoding from a given cardinal
            System.out.println(b64.decodeLong(s[1]));
        }
        if ("-es".equals(s[0])) {
            // generate a b64 encoding from a given string
            System.out.println(b64.encodeString(s[1]));
        }
        if ("-ds".equals(s[0])) {
            // generate a b64 decoding from a given string
            System.out.println(b64.decodeString(s[1]));
        }
        if ("-cl".equals(s[0])) {
            // return the cardinal of a given string as long value with the enhanced encoder
            System.out.println(Base64Order.enhancedCoder.cardinal(s[1].getBytes()));
        }
        if ("-cn".equals(s[0])) {
            // return the cardinal of a given string as normalized float 0 .. 1 with the enhanced encoder
            System.out.println(((double) Base64Order.enhancedCoder.cardinal(s[1].getBytes())) / ((double) Long.MAX_VALUE));
        }
    } */
}


gaston
 
Beiträge: 143
Registriert: Fr Jan 06, 2012 2:22 pm

Re: Speicherverbrauch dauerthaft senken ...

Beitragvon Orbiter » Di Jul 24, 2012 5:04 pm

also dein code ist recht interessant aber die synchronize stören mich sehr, sowas führt dann dazu dass viele Threads dann eben an einer Stelle synchronisieren wo es nicht notwendig wäre. Wie wäre es mit Object pools? Die sind aber auch kostspielig, wobei dann auch nicht klar ist ob die Pooling-Funktionen nicht wieder einen Overhead erzeugen.
Orbiter
 
Beiträge: 5792
Registriert: Di Jun 26, 2007 10:58 pm
Wohnort: Frankfurt am Main

Re: Speicherverbrauch dauerthaft senken ...

Beitragvon Orbiter » Di Jul 24, 2012 5:17 pm

Hier ist auch ein interessanter Artikel zum Thema Garbage Collection:
http://www.ibm.com/developerworks/java/ ... index.html
fazit: nicht darum kümmern. Speicher wird nicht 'verbraucht' bei kurzlebigen Objekten wie ein Integer-Zähler.
Orbiter
 
Beiträge: 5792
Registriert: Di Jun 26, 2007 10:58 pm
Wohnort: Frankfurt am Main

Re: Speicherverbrauch dauerthaft senken ...

Beitragvon gaston » Mi Jul 25, 2012 12:56 pm

Also ein "Object pool" finde ich hier übertrieben. Ich habe das gemacht als ich noch wenig (1200MB) Speicher für YaCy hatte und da finde ich es von Vorteil wen Objekte die immer wieder gebraucht werden, nur 1x angelegt werden. Die synchronize störten und stören mich hier weniger ;)
gaston
 
Beiträge: 143
Registriert: Fr Jan 06, 2012 2:22 pm

Re: Speicherverbrauch dauerthaft senken ...

Beitragvon gaston » Mi Jul 25, 2012 5:55 pm

Unter http://127.0.0.1:8090/IndexControlRWIs_p.html unten die Limitations auf 100000 begrenzen. Vielleicht hilft das nach einer Weile. Ansonsten weniger/langsamer/gar nicht crawlen, weil das verteilen der Daten immer länger dauert als das sammeln.
gaston
 
Beiträge: 143
Registriert: Fr Jan 06, 2012 2:22 pm


Zurück zu Hilfe für Einsteiger und Anwender

Wer ist online?

Mitglieder in diesem Forum: 0 Mitglieder und 1 Gast