Incomplete Wiki !!

Discussion in English language.
Forumsregeln
You can start and continue with posts in english language in all other forums as well, but if you are looking for a forum to start a discussion in english, this is the right choice.

Incomplete Wiki !!

Beitragvon Slntswrd » Mo Apr 13, 2015 3:50 pm

Hi to all!

I'm Reading about this: h t t p://www.yacy-websearch.net/wiki/index.p ... APICrawler
and I have noticed that the explanation in this wiki are incomplete :o !
Can someone of you describe the function of this unlisted parameters?

&createBookmark=on
&bookmarkFolder=/crawlStart
&crawlingIfOlderUnit=hour
&cachePolicy=iffresh
&crawlingIfOlderCheck=on&
bookmarkTitle=&
crawlingDomFilterDepth=1&
crawlingDomFilterCheck=on&
crawlingIfOlderNumber=1

Also can I have an example of Url command for Staring Crawler from: file, url list, and sitemap?
And finally, when i try to start a Crawl process with this kind of cUrl, the best result is a crawling action that only look for the single domain passed for start point (seems like that yacy was set for crawl with deep 1)!

PS
Is this wiki mainteined (last update 2013...)?
Because i can't find an "official" documentation and this lack can be very frustrating :x : !

Cheers!
Slntswrd
 
Beiträge: 4
Registriert: Mi Mär 25, 2015 2:56 pm

Re: Incomplete Wiki !!

Beitragvon biolizard89 » Mo Apr 13, 2015 10:37 pm

I believe that wiki is still maintained to some extent. I am also curious about those missing parameters; I was unaware of them.
biolizard89
 
Beiträge: 58
Registriert: Do Jan 03, 2013 12:42 am

Re: Incomplete Wiki !!

Beitragvon Scarfmonster » Fr Apr 17, 2015 6:53 pm

&createBookmark=on - creates a bookmark at /Bookmarks.html
&bookmarkFolder=/crawlStart - folder/tag to place the bookmark into

&crawlingIfOlderCheck=on
&crawlingIfOlderNumber=1
&crawlingIfOlderUnit=hour

These three are the same as Double-Check Rules in /CrawlStartExpert.html

&cachePolicy=iffresh - this is like Document Cache setting in /CrawlStartExpert.html and is used when &storeHTCache=on

I'm not 100% sure about my knowledge but from my own understanding that's what these do.

As for different crawling modes:
&crawlingMode=url&crawlingURL=
&crawlingMode=sitelist&crawlingURL=
&crawlingMode=sitemap&sitemapURL=
&crawlingMode=file&crawlingfileURL=
Scarfmonster
 
Beiträge: 5
Registriert: Fr Apr 17, 2015 3:20 pm

Re: Incomplete Wiki !!

Beitragvon biolizard89 » Fr Apr 17, 2015 9:59 pm

Scarfmonster hat geschrieben:&createBookmark=on - creates a bookmark at /Bookmarks.html
&bookmarkFolder=/crawlStart - folder/tag to place the bookmark into

&crawlingIfOlderCheck=on
&crawlingIfOlderNumber=1
&crawlingIfOlderUnit=hour

These three are the same as Double-Check Rules in /CrawlStartExpert.html

&cachePolicy=iffresh - this is like Document Cache setting in /CrawlStartExpert.html and is used when &storeHTCache=on

I'm not 100% sure about my knowledge but from my own understanding that's what these do.

As for different crawling modes:
&crawlingMode=url&crawlingURL=
&crawlingMode=sitelist&crawlingURL=
&crawlingMode=sitemap&sitemapURL=
&crawlingMode=file&crawlingfileURL=


How is crawlingIfOlderNumber different from reloadIfOlderNumber?
biolizard89
 
Beiträge: 58
Registriert: Do Jan 03, 2013 12:42 am

Re: Incomplete Wiki !!

Beitragvon Scarfmonster » Sa Apr 18, 2015 12:41 am

biolizard89 hat geschrieben:
Scarfmonster hat geschrieben:&createBookmark=on - creates a bookmark at /Bookmarks.html
&bookmarkFolder=/crawlStart - folder/tag to place the bookmark into

&crawlingIfOlderCheck=on
&crawlingIfOlderNumber=1
&crawlingIfOlderUnit=hour

These three are the same as Double-Check Rules in /CrawlStartExpert.html

&cachePolicy=iffresh - this is like Document Cache setting in /CrawlStartExpert.html and is used when &storeHTCache=on

I'm not 100% sure about my knowledge but from my own understanding that's what these do.

As for different crawling modes:
&crawlingMode=url&crawlingURL=
&crawlingMode=sitelist&crawlingURL=
&crawlingMode=sitemap&sitemapURL=
&crawlingMode=file&crawlingfileURL=


How is crawlingIfOlderNumber different from reloadIfOlderNumber?


Ah sorry, I think the crawlingIf... got renamed to reloadIf, so all of the above are reloadIfOlderCheck etc.
Scarfmonster
 
Beiträge: 5
Registriert: Fr Apr 17, 2015 3:20 pm

Re: Incomplete Wiki !!

Beitragvon Orbiter » Mo Apr 20, 2015 10:33 pm

@Slntswrd sorry if wiki is outdated, but if you find any bug please correct it; thats the purpose of a wiki.
Orbiter
 
Beiträge: 5784
Registriert: Di Jun 26, 2007 10:58 pm
Wohnort: Frankfurt am Main

Re: Incomplete Wiki !!

Beitragvon Slntswrd » Mi Apr 22, 2015 8:50 am

Thanks a lot for the ansewers!
However my second question in the post was:

when I try to start a Crawl process with this kind of cUrl, the best result is a crawling action that only look for the single domain passed for start point (seems like that yacy was set for crawl with deep 1)!


Someone has any ideas?
Have you try to use a post arguments for starting a new crawl? what is the resoult? Maybe the regular expression, or something else, must be Url encoded? or maybe the entire post arguments must be url encoded?

@Orbiter: For be honest, I would do it (updating the wiki) but I don't have even completely undestood the usage of the post arguments API ! Before editing the existing page I Would be sure about what I will have to write in it ;) !
Slntswrd
 
Beiträge: 4
Registriert: Mi Mär 25, 2015 2:56 pm

Re: Incomplete Wiki !!

Beitragvon Scarfmonster » Do Apr 23, 2015 1:21 pm

You need to set the &crawlingDepth= to 2 or 3 or whatever else you want.
Scarfmonster
 
Beiträge: 5
Registriert: Fr Apr 17, 2015 3:20 pm

Re: Incomplete Wiki !!

Beitragvon Slntswrd » Mo Apr 27, 2015 11:32 am

@Scarfmonster:

You are rigth! But I have forgotten to write, that the crawler ALWAYS crawl with depth 1 when I try to start it with post argument!
I had try with all value of &crawlingDepth= from 1 to 9... but nothing change!
Slntswrd
 
Beiträge: 4
Registriert: Mi Mär 25, 2015 2:56 pm

Re: Incomplete Wiki !!

Beitragvon Scarfmonster » Mo Apr 27, 2015 11:41 pm

Could you maybe post an example url you are using to start the crawl?
Scarfmonster
 
Beiträge: 5
Registriert: Fr Apr 17, 2015 3:20 pm


Zurück zu English

Wer ist online?

Mitglieder in diesem Forum: 0 Mitglieder und 1 Gast